Image processing apparatus, image processing method, program, and recording medium

ABSTRACT

There are provided an image processing apparatus, an image processing method, a program, and a recording medium capable of compatibly achieving a high-accuracy filtering process and reduction in a necessary storage capacity. An image processing apparatus includes a filtering process unit that performs an image filtering process including a plurality of filtering processes. The filtering process unit applies a filter to processing target data to acquire filter application process data, and applies a gain to the filter application process data to acquire gain application process data, in each filtering process. In each filtering process, the gain applied to the filter application process data is acquired based on a “target frequency characteristic of the image filtering process” specified based on an optical characteristic of an optical system used when the original image data is acquired, that is, the optical characteristic according to an imaging condition when the original image data is acquired.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of PCT International Application No.PCT/JP2015/073692 filed on Aug. 24, 2015, which claims priority under 35U.S.C. §119 (a) to Japanese Patent Application No. 2014-201084 filed onSep. 30, 2014. Each of the above application(s) is hereby expresslyincorporated by reference, in its entirety, into the presentapplication.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method, a program, and a recording medium, and particularly,to an image filtering process.

2. Description of the Related Art

Various image filtering process techniques such as a filtering processfor enhancing image quality or a filtering process for providing aspecial visual effect have been proposed. For example, a point imagerestoration process that restores an original image using a filterdetermined based on a point spread function (PSF) indicating an imagedeterioration characteristic due to aberration of an imaging lens or thelike is known.

JP2011-124692A discloses an image processing method for performing arestoration process of amplitude components and phase components.According to the image processing method, a restoration strengthadjustment coefficient for adjusting the degree of restoration in therestoration process is set, differential information between a firstimage and a second image, which have the same phase component state buthave different amplitude component states is synthesized with the secondimage according to a restoration strength adjustment coefficient, andthe degree of restoration and the occurrence risk of a adverse effectare adjusted. An image restoration filter used in the restorationprocess is stored in a storage unit, but actually the image restorationfilters stored in the storage unit are discretely arranged in an imagingstate space in which three imaging states of a focal length (state A), adiaphragm value (state B), and a subject distance (state C) are used asaxes.

SUMMARY OF THE INVENTION

If the type and number of filters used in the above-described imagefiltering process increase, a data amount of the entire filtersdramatically increases, and thus, it is necessary to secure a hugestorage area (ROM: read only memory).

That is, in a case where filters used in the image filtering process aredetermined according to a plurality of conditions, a necessary amount ofdata of the filters increases at an exponential rate. For example, apoint spread function of an imaging optical system varies according toimaging conditions such as a diaphragm value (F-number), a zoom value(focal length), a subject distance, a focal position, an optical systemtype, a sensor SN ratio (signal-noise ratio) of an imaging element, animage height (in-image position) and an individual optical systemdifference. However, if characteristic filters are prepared for allcombinations of individual imaging conditions, a total amount of datanecessary for the filters becomes enormous.

Further, in an image filtering process in which specific anisotropy or aspecific frequency characteristic is considered, the number of necessarypatterns of filters increases. Furthermore, in a case where filters(filter coefficients) to be used are changed according to pixelpositions, a data amount of the filters increases according to thenumber of pixels of an image which is a processing target.

Ideally, it is desirable to prepare filters relating to combinations ofall conditions in advance and store the filters in a memory. However,since it is necessary to secure a huge storage capacity and to prepare ahigh-cost memory, it is not necessarily desirable to prepare and storefilters corresponding to all conditions in advance from a viewpoint ofactually building an image processing system.

In the image processing method disclosed in JP2011-124692A, reduction inan amount of stored data is intended by storing the image restorationfilters discretely required with respect to each of a focal length, adiaphragm value, and a subject distance. According to the imageprocessing method, a distance (state difference amount) in imaging statespace between an actual imaging state and a plurality of imaging stateswhere image restoration filters are stored is calculated, and an imagerestoration filter at a position where the distance is the shortest isselected and used in the restoration process. Alternatively, a productof the “distance in imaging state space between an actual imaging stateand a plurality of imaging states where image restoration filters arestored” and a “weighted direction” is set as an evaluation function, andan image restoration filter in which a value of the evaluation functionis the highest is selected and used in the restoration process.Alternatively, an image restoration filter corrected by an interpolationprocess or the like is used in the restoration process.

In the image processing method of JP2011-124692A, as the number of imagerestoration filters to be stored is smaller, a necessary storagecapacity may be further decreased, but a possibility in which an imagerestoration filter which is actually used does not sufficientlycorrespond to an actual imaging state is increased. That is, in a casewhere the image restoration filter corresponding to the actual imagingstate is not stored in the storage unit, an image restoration filtercorresponding to another imaging state is alternatively used, or animage restoration filter corrected and estimated from another imagingstate is used. Therefore, as the number of image restoration filters tobe stored in the storage unit is smaller, a possibility in which adivergence between the image restoration filter which is alternativelyused and the actual imaging state is large is increased, and estimationaccuracy and correction accuracy of the image restoration filter isdeteriorated.

In an image filtering process that uses a multiplicity of filters, thereis a problem that the amount of calculation for determining filtercoefficients increases. For example, in a case where a filter to be usedin an image filtering process is realized by a finite impulse response(FIR) filter having a limited number of taps, the amount of calculationnecessary for computing filter coefficients allocated to each tapincreases.

For example, in a case where a filter is generated based on a pointspread function, a Wiener filter may be calculated from an opticaltransfer function (OTF) obtained by Fourier transforming the pointspread function (PSF). In a case where the number of taps of a filter isnot limited, it is possible to realize a desired frequencycharacteristic obtained by inversely Fourier-transforming the Wienerfilter using a filter having an arbitrary number of taps.

However, if hardware which is actually usable is limited, the number oftaps of a filter is also limited. Thus, in order to calculate a tapcoefficient having a desired frequency characteristic under thecondition that the number of taps of a filter is limited, since it isnecessary to solve a nonlinear optimization problem, a very large amountof calculation is necessary.

Accordingly, in a case where an image filtering process is performedimmediately after an image deterioration characteristic of a pointspread function or the like is acquired, it is necessary to secure atremendous calculation resource for calculation of tap coefficientswhich are actually used. However, it is not necessarily desirable tosecure such a calculation resource from a viewpoint of actually buildingan image processing system.

The invention is made in consideration of the above-described problems,and an object of the invention is to provide an image processingtechnique capable of compatibly achieving a high-accuracy filteringprocess and reduction of a necessary storage capacity.

According to an aspect of the invention, there is provided an imageprocessing apparatus comprising: a filtering process unit that performsan image filtering process that includes a plurality of times offiltering processes with respect to original image data to acquireprocessed image data, in which in each of the plurality of times offiltering processes, the filtering process unit applies a filter toprocessing target data to acquire filter application process data,applies a gain to the filter application process data to acquire gainapplication process data, and acquires filtering process data from thegain application process data, and in which in each of the plurality oftimes of filtering processes, the gain applied to the filter applicationprocess data is acquired based on a target frequency characteristic ofthe image filtering process specified based on an optical characteristicof an optical system used when the original image data is acquired, thatis, an optical characteristic according to an imaging condition when theoriginal image data is acquired.

According to this aspect, in each of the plurality of times of filteringprocesses, the gain applied to the filter application process data isacquired based on the target frequency characteristic of the imagefiltering process, and the target frequency characteristic is specifiedbased on an optical characteristic of an optical system used when theoriginal image data is acquired, that is, an optical characteristicaccording to an imaging condition when the original image data isacquired. By performing the gain adjustment in this way, it is possibleto perform a high-accuracy filtering process according to the imagingcondition while reducing a data amount of a filter applied to processingtarget data in each filtering process.

The “frequency characteristic” is a characteristic relating to afrequency, and represents a ratio of an amplitude for each frequencycomponent of image data after processing to an amplitude for eachfrequency component of image data before processing (=amplitude for eachfrequency component of image data after processing/amplitude for eachfrequency component of image data before processing), and represents aresponse according to a frequency.

Regarding to the optical characteristic, for example, a characteristicbased on a point spread function PSF or an optical transfer function OTF(including a modulation transfer function (MTF) and a phase transferfunction (PTF)) may be used as the “optical characteristic”, and “thetarget frequency characteristic of the image filtering process” may berepresented by using an inverse filter design technique using anarbitrary design standard such as a Wiener filter.

Preferably, the image processing apparatus further comprises: a gaincandidate data storage unit that stores gain table information obtainedby associating candidate data of the gain applied to the filterapplication process data with the imaging condition, in each of theplurality of times of filtering processes; and a gain specifying unitthat specifies, with reference to the gain table information, thecandidate data associated with the imaging condition when the originalimage data is acquired as the gain applied to the filter applicationprocess data in each of the plurality of times of filtering processes.The filtering process unit applies the gain specified by the gainspecifying unit to the filter application process data to acquire thegain application process data in each of the plurality of times offiltering processes.

According to this aspect, it is possible to simply specify the gainapplied to the filter application process data from the gain candidatedata.

The format of “the gain table information” is not particularly limitedas long as gain candidate data and an imaging condition areappropriately associated with each other. The gain table information maydirectly or indirectly associate information (type information of theoptical system, or the like) for specifying the optical system and theimaging condition with the gain candidate data, for example.

Preferably, the filter is determined according to the opticalcharacteristic of the optical system.

According to this aspect, it is possible to perform the filteringprocess with high accuracy based on the optical characteristic of theoptical system.

Preferably, the filter is determined according to the opticalcharacteristic of the optical system under a representative imagingcondition.

According to this aspect, it is possible to perform the filteringprocess of original image data acquired under the representative imagingcondition, with extremely high accuracy.

Here, the “representative imaging condition” is not particularlylimited, and is determined from arbitrary viewpoints. For example, animaging condition discretely selected with respect to each condition,among imaging conditions capable of being employed when the originalimage data is acquired, may be set as a representative imagingcondition, and an imaging condition selected according to a predictedemployed frequency may be set as a representative imaging condition.

Preferably, the representative imaging condition includes a plurality ofimaging conditions in which first index conditions are different, thatis, a plurality of imaging conditions based on the discontinuous firstindex conditions among a plurality of first index conditions capable ofbeing employed in imaging of the original image data.

According to this aspect, the representative imaging condition isdetermined based on the first index condition intermittently selectedamong a plurality of first index conditions capable of being employed inimaging of the original image data.

Here, the “first index condition” is one component constituting therepresentative imaging condition, and is a condition capable of beingchanged in imaging of the original image data. For example, in a casewhere the target frequency characteristic of the image filtering processis determined according to a point spread function, it is possible toemploy one piece of condition data selected among imaging conditionssuch as a diaphragm value (F-number), a zoom value (focal length), asubject distance, a focal position, an optical system type, a sensor SNratio of an imaging element, an image height (in-image position) and anindividual optical system difference, as the first index condition.

Preferably, the optical characteristic is based on the point spreadfunction.

According to this aspect, it is possible to perform the image filteringprocess with high accuracy based on the point spread function.

Preferably, the gain is acquired by fitting a frequency characteristicof the image filtering process to the target frequency characteristic ofthe image filtering process using a least squares method based on eachfrequency characteristic of the plurality of times of filteringprocesses.

According to this aspect, it is possible to acquire a gain for realizingthe target frequency characteristic of the image filtering process withhigh accuracy based on the least squares method.

Preferably, weighting is performed based on a frequency in the leastsquares method.

In particular, in the least squares method, it is preferable that anapproximation error evaluation function of frequency characteristics isweighted based on a frequency. Here, the approximation error evaluationfunction is a function that represents evaluation of the level ofapproximation (that is, the level of an error), and for example, “ageneric function (J_(LMS) [_(g)]) based on a weighted least squaresstandard” which will be described later may be used as the approximationerror evaluation function. According to this aspect, by increasing theweight of a frequency band which is to be emphasized and decreasing theweight in a frequency band which is not to be emphasized, it is possibleto acquire a gain depending on an actual request with high accuracy.

Preferably, a weight in a low-frequency band is set to be larger than aweight in a high-frequency band in the least squares method.

According to this aspect, it is possible to acquire a gain forparticularly realizing a frequency characteristic in a low-frequencyband in the target frequency characteristic of the image filteringprocess with high accuracy. Since a low-frequency component is easilyperceived compared with a high-frequency component in terms of human'svisual characteristics, by increasing the weight in the low-frequencyband to acquire a gain, it is possible to generate processed image datahaving excellent visibility using an image filtering process.

Here, it is preferable that “the low-frequency band” is determinedaccording to an image quality characteristic which is actuallynecessary. For example, the “low-frequency band” may be set in a rangewhere a sampling frequency is equal to or smaller than ¼ (=0.25 fs=½ ofNyquist frequency).

Preferably, a weight in a high-frequency band is set to be larger than aweight in a low-frequency band according to an imaging condition whenthe original image data is acquired, in the least squares method.

According to this aspect, it is possible to acquire a gain forparticularly realizing a frequency characteristic in a high-frequencyband in the target frequency characteristic of the image filteringprocess with high accuracy, according to the imaging condition when theoriginal image data is acquired. Generally, in a case where MTFdeterioration in a high-frequency band is large and there is a largeamount of noise in an imaging system, the noise may be amplified by animage filtering process. That is, image quality may be lowered. Thus, inan image filtering process of original image data acquired under animaging condition that it is predicted that the SN ratio is bad in ahigh-frequency band, it may be preferable to prioritize a high-frequencycomponent with respect to a low-frequency component as the accuracy of“the approximation to the target frequency characteristic of the imagefiltering process”.

Here, it is preferable that “the high-frequency band” is determinedaccording to an image quality characteristic which is actuallynecessary. For example, “the high-frequency band” may be set in a rangewhere a sampling frequency is larger than ¼ (=0.25 fs) and is equal toor smaller than ½ (=0.5 fs), based on a frequency characteristic ofassumed noise.

Further, “the imaging condition when the original image data isacquired” is determined based on an arbitrary factor that may haveinfluences on noise. For example, one or a plurality of conditionsselected from setting conditions of an imaging optical system used whenthe original image data is acquired, subject state conditions (sceneconditions), or the like may be set to “the imaging condition when theoriginal image data is acquired”.

Preferably, the weight in the least squares method is determinedaccording to a pixel position in the original image data.

According to this aspect, it is possible to change the weight in theleast squares method according to the pixel position in the originalimage data, and to perform an image filtering process according to thepixel position. For example, at a pixel position where sharp imagequality is necessary, a weight in a high-frequency band may be set to belarger than a weight in a low-frequency band, and at a pixel positionwhere image quality with excellent visibility is necessary, a weight ina low-frequency band may be set to be larger than a weight in ahigh-frequency band.

Preferably, in the least squares method, the weight in thehigh-frequency band is large at a pixel position which is equal to orshorter than a first distance from the center of an image of theoriginal image data, compared with a pixel position which is moredistant than the first distance from the center of the image of theoriginal image data.

According to this aspect, it is possible to acquire a gain withexcellent reproducibility of a high-frequency component in a centralportion of an image compared with a peripheral portion of the image. Forexample, in a case where a main subject is disposed in the centralportion of the image, or in similar cases, this aspect may be suitablyapplied to a case where sharpness is necessary in the central portion ofthe image.

Here, “the first distance” is not particularly limited, and may beappropriately set based on an image range where a high-frequency band isemphasized.

Preferably, the weight in the low-frequency band is large at a pixelposition which is more distant than a second distance from the center ofan image of the original image data, compared with a pixel positionwhich is equal to or shorter than the second distance from the center ofthe image of the original image data, in the least squares method.

According to this aspect, it is possible to acquire a gain withexcellent reproducibility of a low-frequency component in a peripheralportion of an image compared with a central portion of the image. Forexample, as in a case where a subject which is an observation target maybe disposed in a peripheral portion of an image, in a case whereexcellent visibility is necessary in a peripheral portion of an image,it is possible to suitably apply this aspect.

Here, “the second distance” is not particularly limited, and may beappropriately set based on an image range where a low-frequency band isemphasized.

Preferably, the filtering process unit uses a filter that makes thefiltering process data equal to the processing target data in each ofthe plurality of times of filtering processes at a frequency where aratio of the processed image data to the original image data is smallerthan 1 in the target frequency characteristic of the image filteringprocess.

If an image filtering process is performed using an excessively smallvalue of data, noise is easily mixed in data after processing, and also,such noise may be amplified, which consequently may lead todeterioration in image quality. According to this aspect, by using afilter that makes the filtering process data equal to the processingtarget data, it is possible to effectively prevent mixing of noise andamplification of the noise.

Preferably, the filtering process unit acquires the filter applicationprocess data using a filter having a frequency characteristic accordingto a pixel position in the processing target data, in at least any onefiltering process among the plurality of times of filtering processes.

According to this aspect, the filtering process having the frequencycharacteristic according to the pixel position in the processing targetdata can be performed at least once.

Preferably, the plurality of times of filtering processes include atleast a first filtering process and a second filtering process, and thefiltering process unit uses the filtering process data acquired by thefirst filtering process as the processing target data in the secondfiltering process.

According to this aspect, the first filtering process and the secondfiltering process can be performed in series.

Preferably, the plurality of times of filtering processes include atleast a first filtering process and a second filtering process, and thefiltering process unit uses the same data in the first filtering processand the second filtering process as the processing target data, andacquires the processed image data based on the filtering process dataacquired by the first filtering process and the filtering process dataacquired by the second filtering process.

According to this aspect, the first filtering process and the secondfiltering process can be performed in parallel.

Preferably, the plurality of times of filtering processes include atleast a first filtering process and a second filtering process, and thefiltering process unit includes a first filter application unit thatapplies a filter for the first filtering process to the processingtarget data of the first filtering process to acquire the filterapplication process data, a first gain application unit that applies again for the first filtering process to the filter application processdata acquired by the first filter application unit to acquire the gainapplication process data, a second filter application unit that appliesa filter for the second filtering process to the processing target dataof the second filtering process to acquire the filter applicationprocess data, and a second gain application unit that applies a gain forthe second filtering process to the filter application process dataacquired by the second filter application unit to acquire the gainapplication process data.

According to this aspect, the first filtering process and the secondfiltering process can be performed by “the filter application unit (thefirst filter application unit and the second filter application unit)”and “the gain application unit (the first gain application unit and thesecond gain application unit)” which are separately provided, and thus,the processing flow can become simple.

Preferably, the plurality of times of filtering processes include atleast a first filtering process and a second filtering process, and thefiltering process unit includes a filter application unit that appliesthe filter to the processing target data to acquire the filterapplication process data, and a gain application unit that applies thegain to the filter application process data to acquire the gainapplication process data. The filter application unit acquires thefilter application process data using a filter for the first filteringprocess in the first filtering process, and acquires the filterapplication process data using a filter for the second filtering processin the second filtering process. The gain application unit acquires thegain application process data using a gain for the first filteringprocess in the first filtering process, and acquires the gainapplication process data using a gain for the second filtering processin the second filtering process.

According to this aspect, the first filtering process and the secondfiltering process can be performed by “the filter application unit” and“the gain application unit” which are the same, and thus, a hardwareconfiguration (circuit configuration) can become simple.

Preferably, the plurality of times of filtering processes include atleast a first filtering process and a second filtering process, and thefiltering process unit acquires the filter application process datausing a reference filter determined according to an average of aplurality of types of frequency characteristics of the image filteringprocesses in the first filtering process, and acquires the filterapplication process data using a variance filter determined according toa variance of the plurality of types of frequency characteristics of theimage filtering processes in the second filtering process.

According to this aspect, since the reference filter and the variancefilter determined according to the average and the variance of theplurality of types of frequency characteristics of the image filteringprocesses are used, it is possible to averagely perform an imagefiltering process to be applied with high accuracy.

The variance filter may correspond to directly or indirectly thevariance of the plurality of types of frequency characteristics of theimage filtering processes. For example, the variance filter may bedetermined based on a standard deviation derived from the variance, ormay be determined based on other factors (for example, an average or thelike) in addition to the variance.

According to another aspect of the invention, there is provided an imageprocessing apparatus comprising: a filtering process unit that performsan image filtering process including a plurality of times of filteringprocesses with respect to original image data to acquire processed imagedata. Here, the plurality of times of filtering processes include atleast a first filtering process and a second filtering process. Thefiltering process unit applies a filter to a processing target data toacquire filter application process data, applies a gain to the filterapplication process data to acquire gain application process data, andacquires filtering process data from the gain application process data,in each of the plurality of times of filtering processes, acquires, inthe first filtering process, the filter application process data using areference filter determined according to an average of frequencycharacteristics of the plurality of types of image filtering processes,which are frequency characteristics of the plurality of types of imagefiltering processes determined according to the optical characteristicof the optical system according to an imaging condition when theoriginal image data is acquired, and acquires, in the second filteringprocess, the filter application process data using a variance filterdetermined according to a variance of the frequency characteristics ofthe plurality of types of image filtering processes, which are thefrequency characteristics of the plurality of types of image filteringprocesses determined according to the optical characteristic of theoptical system according to an imaging condition when the original imagedata is acquired. The gain applied to the filter application processdata is determined based on the optical characteristic of the opticalsystem according to an imaging condition when the original image data isacquired, in each of the plurality of times of filtering processes.

According to this aspect, the gain applied to the filter applicationprocess data is determined based on the optical characteristic of theoptical system according to an imaging condition when the original imagedata is acquired, in each of the plurality of times of filteringprocesses. By performing the gain adjustment in this way, it is possibleto realize a desired image filtering process specified based on theoptical characteristic of the optical system according to the imagingcondition with high accuracy using a simple computation process.Further, since the reference filter and the variance filter according tothe average and the variance of the frequency characteristics of theplurality of types of image filtering processes determined based on theoptical characteristic of the optical system according to an imagingcondition are used, it is possible to averagely perform an imagefiltering processes with high accuracy.

According to still another aspect of the invention, there is provided animage processing method for performing an image filtering processincluding a plurality of times of filtering processes with respect tooriginal image data to acquire processed image data, comprising:applying a filter to processing target data to acquire filterapplication process data, applying a gain to the filter applicationprocess data to acquire gain application process data, and acquiringfiltering process data from the gain application process data, in eachof the plurality of times of filtering processes, and acquiring the gainapplied to the filter application process data based on a targetfrequency characteristic of the image filtering process specified basedon an optical characteristic of an optical system used when the originalimage data is acquired, that is, the optical characteristic according toan imaging condition when the original image data is acquired, in eachof the plurality of times of filtering processes.

According to still another aspect of the invention, there is provided animage processing method for performing an image filtering processincluding a plurality of times of filtering processes with respect tooriginal image data to acquire processed image data, in which theplurality of times of filtering processes include at least a firstfiltering process and a second filtering process, in which the methodcomprises: applying a filter to processing target data to acquire filterapplication process data, applying a gain to the filter applicationprocess data to acquire gain application process data, and acquiringfiltering process data from the gain application process data, in eachof the plurality of times of filtering processes; acquiring, in thefirst filtering process, the filter application process data using areference filter determined according to an average of frequencycharacteristics of a plurality of types of image filtering processes,which are frequency characteristics of the plurality of types of imagefiltering processes determined according to the optical characteristicof the optical system according to an imaging condition when theoriginal image data is acquired; and acquiring, in the second filteringprocess, the filter application process data using a variance filterdetermined according to a variance of the frequency characteristics ofthe plurality of types of image filtering processes, which are thefrequency characteristics of the plurality of types of image filteringprocesses determined according to the optical characteristic of theoptical system according to an imaging condition when the original imagedata is acquired, and in which the gain applied to the filterapplication process data is determined based on the opticalcharacteristic of the optical system according to an imaging conditionwhen the original image data is acquired, in each of the plurality oftimes of filtering processes.

According to still another aspect of the invention, there is provided aprogram that causes a computer to realize a function for performing animage filtering process including a plurality of times of filteringprocesses with respect to original image data to acquire processed imagedata, the program causing the computer to execute: a process of applyinga filter to processing target data to acquire filter application processdata, a process of applying a gain to the filter application processdata to acquire gain application process data, and a process ofacquiring filtering process data from the gain application process data,in each of the plurality of times of filtering processes, in which thegain applied to the filter application process data is acquired based ona target frequency characteristic of the image filtering processspecified based on an optical characteristic of an optical system usedwhen the original image data is acquired, that is, the opticalcharacteristic according to an imaging condition when the original imagedata is acquired, in each of the plurality of times of filteringprocesses.

According to still another aspect of the invention, there is provided aprogram that causes a computer to realize a function for performing animage filtering process including a plurality of times of filteringprocesses with respect to original image data to acquire processed imagedata, in which the plurality of times of filtering processes include atleast a first filtering process and a second filtering process, in whichthe program causes the computer to execute: a process of applying afilter to processing target data to acquire filter application processdata, a process of applying a gain to the filter application processdata to acquire gain application process data, and a process ofacquiring filtering process data from the gain application process data,in each of the plurality of times of filtering processes; a process ofacquiring, in the first filtering process, the filter applicationprocess data using a reference filter determined according to an averageof frequency characteristics of the plurality of types of imagefiltering processes, which are the frequency characteristics of theplurality of types of image filtering processes determined according tothe optical characteristic of the optical system according to an imagingcondition when the original image data is acquired; and a process ofacquiring, in the second filtering process, the filter applicationprocess data using a variance filter determined according to a varianceof the frequency characteristics of the plurality of types of imagefiltering processes, which are the frequency characteristics of theplurality of types of image filtering processes determined according tothe optical characteristic of the optical system according to an imagingcondition when the original image data is acquired, and in which thegain applied to the filter application process data is determined basedon the optical characteristic of the optical system according to animaging condition when the original image data is acquired, in each ofthe plurality of times of filtering processes.

According to still another aspect of the invention, there is provided acomputer-readable recording medium that stores a program that causes acomputer to realize a function for performing an image filtering processincluding a plurality of times of filtering processes with respect tooriginal image data to acquire processed image data, in which theprogram causes the computer to execute: a process of applying a filterto processing target data to acquire filter application process data, aprocess of applying a gain to the filter application process data toacquire gain application process data, and a process of acquiringfiltering process data from the gain application process data, in eachof the plurality of times of filtering processes, and in which the gainapplied to the filter application process data is acquired based on atarget frequency characteristic of the image filtering process specifiedbased on an optical characteristic of an optical system used when theoriginal image data is acquired, that is, the optical characteristicaccording to an imaging condition when the original image data isacquired, in each of the plurality of times of filtering processes.

According to still another aspect of the invention, there is provided acomputer-readable recording medium that stores a program that causes acomputer to realize a function for performing an image filtering processincluding a plurality of times of filtering processes with respect tooriginal image data to acquire processed image data, in which theplurality of times of filtering processes include at least a firstfiltering process and a second filtering process, in which the programcauses the computer to execute: a process of applying a filter toprocessing target data to acquire filter application process data, aprocess of applying a gain to the filter application process data toacquire gain application process data, and a process of acquiringfiltering process data from the gain application process data, in eachof the plurality of times of filtering processes; a process ofacquiring, in the first filtering process, the filter applicationprocess data using a reference filter determined according to an averageof frequency characteristics of the plurality of types of imagefiltering processes, which are the frequency characteristics of theplurality of types of image filtering processes determined according tothe optical characteristic of the optical system according to an imagingcondition when the original image data is acquired; and a process ofacquiring, in the second filtering process, the filter applicationprocess data using a variance filter determined according to a varianceof the frequency characteristics of the plurality of types of imagefiltering processes, which are the frequency characteristics of theplurality of types of image filtering processes determined according tothe optical characteristic of the optical system according to an imagingcondition when the original image data is acquired, and in which thegain applied to the filter application process data is determined basedon the optical characteristic of the optical system according to animaging condition when the original image data is acquired, in each ofthe plurality of times of filtering processes.

According to the invention, a gain applied to filter application processdata in each of a plurality of times of filtering processes is acquiredbased on a target frequency characteristic of an image filteringprocess, and the target frequency characteristic is specified based onan optical characteristic of an optical system used when the originalimage data is acquired, that is, an optical characteristic according toan imaging condition when the original image data is acquired. Thus, itis possible to perform a high-accuracy filtering process according tothe imaging condition while reducing a data amount of a filter appliedto processing target data in each filtering process.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a digital camera connected to acomputer.

FIG. 2 is a block diagram showing an example of a configuration of amain body controller.

FIG. 3 is a block diagram showing an example of a functionalconfiguration of an image processing unit.

FIG. 4 is a block diagram showing another example of the functionalconfiguration of the image processing unit.

FIG. 5 is a block diagram showing another example of the functionalconfiguration of the image processing unit.

FIG. 6 is a conceptual diagram of an image filtering process.

FIG. 7 is a conceptual diagram of an n-th filtering process (here, “n”is an integer which is equal to or greater than 1 and is equal to orsmaller than N).

FIG. 8 is a diagram showing an example of a relationship between anF-number capable of being employed in imaging of the original image dataand a representative F-number.

FIG. 9 is a diagram showing an example of the circuit configuration ofthe filtering process unit.

FIG. 10 is a diagram showing another example of the circuitconfiguration of the filtering process unit.

FIG. 11 is a diagram showing another example of the circuitconfiguration of the filtering process unit.

FIG. 12 is a diagram showing another example of the circuitconfiguration of the filtering process unit.

FIG. 13 is a block diagram showing an example of a functionalconfiguration of a gain acquisition unit, and shows a gain acquisitionunit which is suitably combined with the image processing unit shown inFIG. 3.

FIG. 14 is a flowchart showing a gain acquisition flow of the gainacquisition unit shown in FIG. 13.

FIG. 15 is a block diagram showing another example of the functionalconfiguration of the gain acquisition unit, and shows a gain acquisitionunit which is suitably combined with the image processing unit shown inFIGS. 4 and 5.

FIG. 16 is a flowchart showing a frequency characteristic acquisitionflow of a frequency analysis unit shown in FIG. 15.

FIG. 17 is a flowchart showing a gain acquisition flow of the gainacquisition unit shown in FIG. 15.

FIG. 18 is a block diagram showing an example of a functionalconfiguration of a filter acquisition apparatus.

FIG. 19 is a block diagram showing an example of a functionalconfiguration of a filter calculation unit.

FIG. 20 is a conceptual diagram representing variation in frequencycharacteristics of a plurality of types of image filtering processes asa unimodal distribution.

FIG. 21 is a conceptual diagram representing variation in frequencycharacteristics of a plurality of types of image filtering processes asa multimodal distribution.

FIG. 22 is a diagram schematically showing an example of frequencycharacteristics of a plurality of types of image filtering processescapable of being classified as a unimodal distribution, in which alateral axis represents a frequency and a longitudinal axis represents aresponse.

FIG. 23 is a diagram schematically showing an example of a variance offrequency characteristics of the plurality of types of image filteringprocesses shown in FIG. 22, in which a lateral axis represents afrequency and a longitudinal axis represents a variance.

FIGS. 24A to 24C are diagrams schematically showing examples of basefilters (characteristic vectors) acquired from an average and a varianceof frequency characteristics of the plurality of types of imagefiltering processes shown in FIG. 22, in which FIG. 24A shows “φ₀”, FIG.24B shows “φ₁”, and FIG. 24C shows “φ₂”.

FIGS. 25A and 25B show a frequency characteristic (see FIG. 25A) and again example (see FIG. 25B) relating to image filtering processes (firstfiltering process to third filtering process) performed by a filteringprocess unit having the circuit configuration shown in FIG. 11.

FIG. 26 is a block diagram showing an example of a functionalconfiguration of a filter calculation unit that calculates a filtercoefficient based on a unimodal distribution.

FIG. 27 is a diagram schematically showing an example of frequencycharacteristic of image filtering processes capable of being classifiedas a multimodal distribution, in which a lateral axis represents afrequency and a longitudinal axis represents a response.

FIGS. 28A and 28B are diagrams schematically showing an example of avariance of frequency characteristics of a plurality of types of imagefiltering processes shown in FIG. 27, in which FIG. 28A shows a varianceof frequency characteristic data of a first image filtering process to athird image filtering process classified as a first variationdistribution, and FIG. 28B shows a variance of frequency characteristicdata of a fourth image filtering process to a sixth image filteringprocess classified as a second variation distribution.

FIGS. 29A to 29C are diagrams schematically showing examples of basefilters (characteristic vectors) acquired from an average and a varianceof frequency characteristics of the first image filtering process to thethird image filtering process classified as the first variationdistribution, in which FIG. 29A shows “φ₀”, FIG. 29B shows “φ₁”, andFIG. 29C shows “φ₂”.

FIGS. 30A to 30C are diagrams schematically showing examples of basefilters (characteristic vectors) acquired from an average and a varianceof frequency characteristics of the fourth image filtering process tothe sixth image filtering process classified as the second variationdistribution, in which FIG. 30A shows “φ₀”, FIG. 30B shows “φ₁”, andFIG. 30C shows “φ₂”.

FIGS. 31A and 31B show a frequency characteristic (see FIG. 31A) and again example (see FIG. 31B) relating to image filtering processes (firstfiltering process to third filtering process) performed by a filteringprocess unit having the circuit configuration shown in FIG. 11.

FIG. 32 is a block diagram showing an example of a functionalconfiguration of a filter calculation unit that calculates a filtercoefficient based on a multimodal distribution.

FIG. 33 is a conceptual diagram of a plurality of types of imagefiltering processes (frequency characteristics) classified as aplurality of filter groups by a filter group classification unit.

FIG. 34 is a block diagram showing a form of an imaging module thatincludes an EDoF optical system.

FIG. 35 is a diagram showing an example of the EDoF optical system.

FIG. 36 is a diagram showing an example of a restoration processing flowin a restoration process block shown in FIG. 34.

(a) and (b) of FIG. 37 are diagrams showing an example of restoration ofan image acquired through the EDoF optical system, in which (a) of FIG.37 shows a blurred image before the restoration process and (b) of FIG.37 shows an image (point image) in which blurring is canceled after therestoration process.

FIG. 38 is a diagram showing an appearance of a smartphone.

FIG. 39 is a block diagram showing a configuration of the smartphoneshown in FIG. 38.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the invention will be described with reference to theaccompanying drawings. In the following embodiments, a case in which theinvention is applied to a digital camera (imaging apparatus) capable ofbeing connected to a computer (PC: personal computer) will be describedas an example.

FIG. 1 is a block diagram showing a digital camera 10 connected to acomputer 92.

The digital camera 10 in this example includes an exchangeable lens unit11, and a camera main body 12 that includes an imaging element 24, inwhich the lens unit 11 and the camera main body 12 are electricallyconnected to each other through a lens unit input/output unit 20 of thelens unit 11 and a camera main body input/output unit 28 of the cameramain body 12.

The lens unit 11 includes an optical system that includes a lens 14 anda diaphragm 15, and an optical system operation unit 17 that controlsthe optical system. The optical system operation unit 17 includes a lensunit controller 18 connected to the lens unit input/output unit 20, alens unit storage unit 19 that stores a variety of information such asoptical system information, and an actuator (not shown) that operatesthe optical system. The lens unit controller 18 controls the opticalsystem through the actuator based on a control signal transmitted fromthe camera main body 12 through the lens unit input/output unit 20, andfor example, performs a focus control or a zoom control based on lensmovement, a diaphragm amount control of the diaphragm 15, and the like.Further, the lens unit controller 18 reads out a variety of informationstored in the lens unit storage unit 19 based on a control signaltransmitted from the camera main body 12 through the lens unitinput/output unit 20, and transmits the read information to the cameramain body 12 (main body controller 25).

The imaging element 24 of the camera main body 12 includes a focusingmicrolens, color filters of R (red), G (green) and B (blue), or thelike, and an image sensor (photodiode) configured by a complementarymetal oxide semiconductor (CMOS), a charge coupled device (CCD) or thelike. The imaging element 24 converts subject image light irradiatedthrough the optical system (the lens 14, the diaphragm 15, and the like)of the lens unit 11 into an electric signal, and transmits the imagesignal (image data) to the main body controller 25.

The main body controller 25 has a function as a device controller thatgenerally controls the respective units of the digital camera 10, and afunction as an image processing unit (image processing apparatus) thatperforms image processing with respect to image data transmitted fromthe imaging element 24, but details thereof will be described later (seeFIG. 2).

The digital camera 10 includes other devices (release button or thelike) necessary for imaging or the like, and a part of the other devicesforms a user interface 26 through which a user is capable of performingchecks and operations. In the example shown in FIG. 1, the userinterface 26 is provided in the camera main body 12, but the userinterface 26 may be disposed in the lens unit 11 and/or the camera mainbody 12. The user may determine and change various settings (exposurevalue (EV)) for imaging or the like, may give an imaging instruction, ormay check a live view image and a captured image through the userinterface 26, for example. The user interface 26 is connected to themain body controller 25, so that various setting and variousinstructions determined and changed by the user are reflected in variousprocesses in the main body controller 25. The user is guided through adisplay on a display unit 27 (electronic view finder (EVF), and a rearsurface liquid crystal display unit) included in the user interface 26,and thus, imaging may be prompted.

Image data which is subjected to image processing in the main bodycontroller 25 is stored in a main body storage unit 29 provided in thecamera main body 12, and is transmitted to an external apparatus such asthe computer 92 through an input/output interface 30, as necessary. Themain body storage unit 29 is configured by an arbitrary memory device,and an exchangeable memory such as a memory card may be suitably used. Aformat of the image data output from the main body controller 25 is notparticularly limited, and image data having a format such as RAW, jointphotographic experts group (JPEG) and/or tagged image file format (TIFF)may be generated and output by the main body controller 25. Further, themain body controller 25 may form one image file by associating aplurality of pieces of related data such as header information (imaginginformation (imaging date and time, device type, the number of pixels, adiaphragm value, or the like), or the like), main image data andthumbnail image data with each other, such as a so-called exchangeableimage file format (Exif), and may output the image file.

The computer 92 is connected to the digital camera 10 through theinput/output interface 30 of the camera main body 12 and the computerinput/output unit 93, and receives a variety of data such as image datatransmitted from the camera main body 12. The computer controller 94totally controls the computer 92, performs image processing with respectto image data from the digital camera 10, and for example, controlscommunication with a server 97 connected to the computer input/outputunit 93 through a network 96 such as the Internet. The computer 92includes a display 95, and processed content or the like in the computercontroller 94 is displayed on the display 95 as necessary. A user mayoperate input means (not shown) such as a keyboard while checking adisplay on the display 95, to thereby input data or commands into thecomputer controller 94. Thus, the user may control the computer 92 ordevices (the digital camera 10, the server 97, and the like) connectedto the computer 92.

The server 97 includes a server input/output unit 98 and a servercontroller 99. The server input/output unit 98 configures atransmission/reception connection unit with respect to an externalapparatus such as the computer 92, and is connected to the computerinput/output unit 93 of the computer 92 through the network 96 such asthe Internet. The server controller 99 works in cooperation with thecomputer controller 94 according to a control instruction signal fromthe computer 92, performs transmission and reception of a variety ofdata with respect to the computer controller 94 as necessary, performs acomputation process, and transmits a computation result to the computer92.

Each controller (the lens unit controller 18, the main body controller25, the computer controller 94, and the server controller 99) includes avariety of circuits necessary for a control process, and for example,includes a computational circuit (central processing unit (CPU), or thelike), a memory, or the like. Further, communication between the digitalcamera 10, the computer 92, and the server 97 may be a wiredcommunication or a wireless communication. In addition, the computer 92and the server 97 may be integrally formed, or the computer 92 and/orthe server 97 may not be provided. Furthermore, a communication functionwith respect to the server 97 may be given to the digital camera 10, andtransmission and reception of a variety of data may be directlyperformed between the digital camera 10 and the server 97.

FIG. 2 is a block diagram showing an example of a configuration of themain body controller 25. The main body controller 25 includes a devicecontroller 34 and an image processing unit (image processing apparatus)35, and totally controls the camera main body 12.

The device controller 34 appropriately controls various types of devicesprovided in the digital camera 10, for example, controls the imagingelement 24 to control an output of an image signal (image data) from theimaging element 24, generates a control signal for controlling the lensunit 11, transmits the control signal to the lens unit 11 (lens unitcontroller 18) through camera main body input/output unit 28, stores theimage data before and after image processing (RAW data, JPEG data, orthe like) in the main body storage unit 29, and transmits the image databefore and after the image processing (RAW data, JPEG data, or the like)to an external apparatus or the like (computer 92 or the like) connectedthrough the input/output interface 30. Further, the device controller 34of this example includes a display controller 36 that controls thedisplay unit 27. The display controller 36 may cause the display unit 27to display a guide for guiding a user's imaging operation in cooperationwith the image processing unit 35.

On the other hand, the image processing unit 35 performs arbitrary imageprocessing with respect to an image signal output from the imagingelement 24, as necessary. For example, a variety of image processingsuch as sensor correction, demosaicing (synchronization), pixelinterpolation, color correction (offset correction, white balancing,color matrix processing, gamma conversion, and the like), and RGB imageprocessing (sharpening, tone correction, exposure correction, contourcorrection, and the like), RGB/YCrCb conversion, or image compressionmay be appropriately performed in the image processing unit 35.

FIG. 3 is a block diagram showing an example of a functionalconfiguration of the image processing unit 35.

The image processing unit 35 includes a filtering process unit 41 thatperforms an image filtering process including a plurality of times offiltering processes with respect to original image data D1 to acquireprocessed image data D2. Further, the image processing unit 35 of thisexample further includes a gain specifying unit 43, a gain candidatedata storage unit 44, a pre-processing unit 40, and a post-processingunit 42, in addition to the filtering process unit 41.

The filtering process unit 41 applies a filter to processing target datato acquire filter application process data in each of the plurality oftimes of filtering processes, applies a gain to the filter applicationprocess data to acquire gain application process data, and acquiresfiltering process data from the gain application process data.

The gain candidate data storage unit 44 stores “gain table information”obtained by associating candidate data of a gain applied to filterapplication process data in each of the plurality of times of filteringprocesses in the filtering process unit 41 with an imaging condition.The gain table information in this example is configured so that thecandidate data of the gain is associated with the type of the opticalsystem and the imaging condition. The gain table information includingthe candidate data of the gain is acquired in advance by the gainacquisition unit 45 according to the type of the optical system and theimaging condition, and is stored in the gain candidate data storage unit44. A specific example of a gain (candidate data) acquisition methodbased on the type of the optical system and the imaging condition willbe described later.

The gain specifying unit 43 specifies, with reference to the gain tableinformation stored in the gain candidate data storage unit 44, candidatedata associated with an imaging condition when the original image dataD1 is acquired as a gain to be applied to the filter application processdata in each of the plurality of times of filtering processes, andtransmits the result to the filtering process unit 41. The gainspecifying unit 43 in this example acquires information about the typeof an optical system used when the original image data D1 which is atarget of the image filtering process is acquired and an imagingcondition, reads out the candidate data of the gain associated with thetype of the optical system and the imaging condition from the gaincandidate data storage unit 44, and transmits the result to thefiltering process unit 41 as a gain to be applied to the filterapplication process data. A method for acquiring information about thetype of the optical system and the imaging condition in the gainspecifying unit 43 is not particularly limited, and the gain specifyingunit 43 may acquire information about the type of the optical system andthe imaging condition from information about the lens unit 11 andimaging condition information retained in a memory (not shown) of themain body controller 25.

The filtering process unit 41 applies the gain specified by the gainspecifying unit 43 in each of the plurality of times of filteringprocesses to the filter application process data to acquire the gainapplication process data.

The image processing unit 35 also performs image processing other thanthe above-described image filtering process. A variety of imageprocessing other than the image filtering process may be performed inthe pre-processing unit 40 provided at a pre-stage of the filteringprocess unit 41, or may be performed in the post-processing unit 42provided at a post-stage of the filtering process unit 41. That is, theimage filtering process in the filtering process unit 41 may beperformed using data received through the variety of image processingperformed in the pre-processing unit 40 as the original image data D1,or the variety of image processing may be performed in thepost-processing unit 42 with respect to the processed image data D2received through the image filtering process in the filtering processunit 41.

In this example, image data input to the filtering process unit 41 fromthe pre-processing unit 40 is referred to as the “original image dataD1”, and image data output from the post-processing unit 42 from thefiltering process unit 41 is referred to as the “processed image dataD2”.

FIG. 4 is a block diagram showing another example of the functionalconfiguration of the image processing unit 35.

In the example shown in FIG. 3, a gain to be used in the filteringprocess is acquired with reference to the gain table information storedin advance in the gain candidate data storage unit 44, but in theexample shown in FIG. 4, the gain table information (gain candidatedata) is not acquired and stored in advance, and instead, a gain isacquired together with execution of the filtering process. That is, inthe example shown in FIG. 4, the gain acquisition unit 45 is connectedto the filtering process unit 41, and a gain to be used in the pluralityof times of filtering processes performed in the filtering process unit41 is directly supplied to the filtering process unit 41 from the gainacquisition unit 45. The gain acquisition unit 45 acquires dataindicating the optical characteristic of the optical system used whenthe original image data D1 is acquired, that is, the opticalcharacteristic according to the imaging condition when the originalimage data D1 is acquired, specifies a target frequency characteristicof the image filtering process based on the data indicating the opticalcharacteristic, acquires a gain to be applied to the filter applicationprocess data in each of the plurality of times of filtering processes,based on the specified “target frequency characteristic of the imagefiltering process”, and supplies the acquired gain to the filteringprocess unit 41.

Here, as the data indicating the “optical characteristic of the opticalsystem used when the original image data D1 is acquired”, data (forexample, a model number or the like) indicating the type of the lensunit 11 (the optical system) used when the original image data D1 isacquired through imaging is exemplified. Further, the “data indicatingthe optical characteristic according to an imaging condition when theoriginal image data D1 is acquired” is data (for example, F-number orthe like) indicating an imaging condition when the original image dataD1 is acquired, for example. A method for acquiring the data based onthe gain acquisition unit 45 is not particularly limited, and the gainacquisition unit 45 may acquire data of the type of the optical systemand the imaging condition from the information about the lens unit 11and the imaging condition information retained in a memory (not shown)of the main body controller 25, for example.

The gain acquisition unit 45 may be provided as a part of the imageprocessing unit 35, as shown in FIG. 4, or may be provided as anotherprocessing unit which is different from the image processing unit 35, asshown in FIG. 5.

In the examples shown in FIGS. 4 and 5, similarly, image processingother than the above-described image filtering process may be performedin the pre-processing unit 40 provided at the pre-stage of the filteringprocess unit 41, or may be performed in the post-processing unit 42provided at the post-stage of the filtering process unit 41.

Next, a specific example of the image filtering process (the pluralityof times of filtering processes) performed in the filtering process unit41 will be descried.

In a case where a filter used in the filtering process is determinedbased on the optical characteristic of the optical system, in a generalimage filtering process, when an “optical characteristic of the opticalsystem under a first imaging condition” and an “optical characteristicof the optical system under a second imaging condition” are different,it is preferable that a filter according to the first imaging conditionis used in a filtering process of an image captured under the firstimaging condition, and a filter according to the second imagingcondition is used in a filtering process of an image captured under thesecond imaging condition. In this case, it is necessary to prepare thefilter for each imaging condition, but in a case where there is aplurality of factors that affects the optical characteristic of theoptical system as the imaging condition, the number of filters to beprepared and a data amount thereof become enormous. For example, in acase where the optical characteristic described above is based on apoint spread function and a filtering process based on the point spreadfunction is performed, since the point spread function may be changed byimaging conditions such as an F-number, a zoom value, a subjectdistance, a focal position, an optical system type, a sensor SN ratio ofthe imaging element 24, an image height (an in-image position), or anindividual optical system difference, ideally, it is necessary toprepare filters (tap coefficients) corresponding to all combinations ofthe imaging conditions.

In order to prevent such an increase in the amount of data necessary forthe image filtering process, in the following embodiment, instead ofretaining filters corresponding to all combinations of imagingconditions, by applying a plurality of times of filtering processes andadjusting a gain used in each filtering process according to the imagingconditions, an image filtering process suitable for imaging conditionsof original image data which is a processing target is realized.

FIG. 6 is a conceptual diagram of an image filtering process P. Theimage filtering process P of this example includes a plurality of timesof filtering processes (a first filtering process F1 to an N-thfiltering process FN (here, “N” is an integer which is equal to orgreater than 2)). The processed image data D2 is generated from theoriginal image data D1 using the “image filtering process P includingthe plurality of times of filtering processes”.

FIG. 7 is a conceptual diagram of an n-th filtering process Fn (here,“n” is an integer which is equal to or greater than 1 and is equal to orsmaller than N). Each filtering process (the n-th filtering process Fn)of the image filtering process P includes a filter application processedPF and a gain application process PG. The filter application process PFis a process of applying a filter to processing target data D3 toacquire filter application process data D4, and the gain applicationprocess PG is a process of applying a gain to the filter applicationprocess data D4 to acquire gain application process data D5.

In each of the plurality of times of filtering processes (the firstfiltering process F1 to the N-th filtering process FN) included in theimage filtering process P performed by the filtering process unit 41, afilter applied to the processing target data D3 in the filterapplication process PF is not particularly limited.

For example, the filter used in the filter application process PF may bedetermined according to an optical characteristic of an optical system,and a “point spread function” of the optical system may be suitably usedas the “optical characteristic of the optical system” used as the basisof the filter. By determining the filter according to the opticalcharacteristic of the optical system, it is possible to accuratelyperform the image filtering process P (filtering processes) of theoriginal image data D1 influenced by the optical characteristic of theoptical system. For example, by using a filter designed based on thepoint spread function, it is possible to perform the image filteringprocess P (filtering processes) for reducing the influence of the pointspread phenomenon.

The “filter determined according to the optical characteristic of theoptical system” is not limited to the filter based on the point spreadfunction, and a contour emphasis filter or another filter may be used asthe “filter determined according to the optical characteristic of theoptical system”. Even in the case of the filter (contour emphasis filteror the like) other than the filter based on the point spread function,the filter (contour emphasis filter or the like) may be adjustedaccording to various conditions (for example, the above-describedvarious factors such as a diaphragm value (F-number)), and a filter suchas a contour emphasis filter having a filter coefficient (tapcoefficient) adjusted based on optical characteristics that changesaccording to the various conditions (for example, the above-describedvarious factors such as a diaphragm value (F-number)) may be used ineach filter application process PF.

Accordingly, in at least any one filtering process among the pluralityof times of filtering processes (the first filtering process F1 to theN-th filtering process FN), the filtering process unit 41 may acquirefilter application process data D4 using a filter determined accordingto an optical characteristic (point spread function, and the like) ofthe optical system (lens 14, diaphragm 15, and the like).

Further, in a case where a filter is determined according to an opticalcharacteristic of an optical system, in a case where the opticalcharacteristic of the optical system varies according to imagingconditions, the filter may be determined according to the opticalcharacteristic of the optical system under a representative imagingcondition. For example, in a case where a filter is determined accordingto a point spread function of an optical system, it is possible todesign the filter based on the point spread function under arepresentative condition with respect to one or a plurality of imagingconditions among imaging conditions such as an F-number, a zoom value, asubject distance, a focal position, an optical system type, a sensor SNratio of the imaging element 24 which can influence the point spreadfunction, an image height (in-image position) or an individual opticalsystem difference. By designing and storing the filter according to theoptical characteristic of the optical system in the representativeimaging condition, it is possible to save a necessary storage capacity.In an image filtering process P of original image data D1 acquired underimaging conditions other than the “representative imaging condition”, byadjusting a gain, it is possible to perform a filtering process withhigh accuracy, as described below.

In this case, it is preferable that the representative imaging conditionincludes a plurality of imaging conditions in which first indexconditions are different, that is, a plurality of imaging conditionsbased on the discontinuous first index conditions among a plurality offirst index conditions capable of being employed in imaging of theoriginal image data. For example, in a case where a filter is determinedaccording to a point spread function of an optical system, any onecomponent selected among imaging conditions such as an F-number, a zoomvalue, a subject distance, a focal position, an optical system type, asensor SN ratio of an imaging element 24, an image height (in-imageposition), and an individual optical system difference that may affectthe point spread function may be set as the first index condition.

FIG. 8 is a diagram showing an example of a relationship between anF-number capable of being employed in imaging of the original image dataand a representative F-number. For example, in a case where a filter isdetermined according to a point spread function of an optical system,when an F-number is changed as an imaging condition, in general, thepoint spread function of the optical system is also changed. In a casewhere there is a plurality of F-numbers (in the example shown in FIG. 8,a total of 12 from F1 to F45) capable of being employed in imaging ofthe original image data as an imaging condition (a first indexcondition), a filter may be designed based on a point spread functioncorresponding a representative F-number selected among these employableF-numbers. As shown in an example of FIG. 8, a filter may be designed byusing F-numbers (in the example shown in FIG. 8, F1.4, F2.8, F5.6, F11,F22, and F45) alternately (intermittently) and sequentially selectedamong employable F-numbers as the “representative imaging condition”.For example, “the representative imaging condition” may be determinedbased on a predicted “employed frequency when the original image data D1is acquired”, or the filter may be designed using an imaging conditionhaving a high predicted employed frequency as the “representativeimaging condition”.

Accordingly, the filtering process unit 41 may use a plurality offilters based on the point spread function of the optical system (thelens 14, the diaphragm 15, and the like), which is a plurality offilters obtained by changing factors affecting a characteristic of thepoint spread function, in the plurality of times of filtering processes(the first filtering process F1 to the N-th filtering process FN). Asthe factors that affect the characteristic of the point spread function,for example, imaging conditions such as a diaphragm value (F-number), azoom value (focal length), a subject distance, a focal position, anoptical system type, a sensor SN ratio of the imaging element 24, animage height (in-image position), or an individual optical systemdifference may be used. Filters derived based on point spread functionshaving different characteristic data in one or a plurality of factorsamong these factors may be used in each filter application process PF.

Further, the filtering process unit 41 may acquire the filterapplication process data D4 using a filter determined irrespective ofthe characteristics of the optical system (the lens 14, the diaphragm15, and the like) used when the original image data D1 is acquired byimaging, in at least any one filtering process among the plurality oftimes of filtering processes (the first filtering process F1 to the N-thfiltering process FN). As the “filter determined irrespective of thecharacteristics of the optical system”, a contour emphasis filter may besuitably used, for example.

In addition, the filter used in the filter application process PF may bechanged based on a pixel position of the processing target data D3. Thefiltering process unit 41 may acquire the filter application processdata D4 using a filter having a frequency characteristic according tothe pixel position in the processing target data D3, in at least onefiltering process among the plurality of times of filtering processes(the first filtering process F1 to the N-th filtering process FN).Further, without depending on the pixel position of the processingtarget data D3, the same filter may be applied to the processing targetdata D3. The filtering process unit 41 may acquire the filterapplication process data D4 using a filter having a frequencycharacteristic irrespective of the pixel position in the processingtarget data D3, in at least any one filtering process among theplurality of times of filtering processes.

On the other hand, in each of the plurality of times of filteringprocesses (the first filtering process F1 to the N-th filtering processFN) in the image filtering process P, a gain applied to the filterapplication process data D4 in the gain application process PG isacquired based on the “target frequency characteristic of the imagefiltering process P specified based on an optical characteristic of anoptical system used when the original image data is acquired, that is,an optical characteristic according to an imaging condition when theoriginal image data is acquired”.

The “target frequency characteristic of the image filtering process P”may be experimentally or theoretically obtained in advance according tothe type of the optical system, and the “gain used in the gainapplication process PG” acquired based on the “target frequencycharacteristic of the image filtering process P” may also be obtained inadvance.

For example, in a case where the “point spread function” is assumed asthe optical characteristic and the “F-number” is assumed as an exampleof imaging conditions when the original image data D1 is acquired, anF-number capable of being employed when the original image data D1 isacquired through imaging may be assumed in advance, and a point spreadfunction of the optical system corresponding to each of the “F-numbercapable of being employed” may be derived in advance. The targetfrequency characteristic of the image filtering process P may berepresented by a frequency characteristic based on the point spreadfunction (optical transfer function) of the optical system, and aninverse filter design method using an arbitrary design standard such asa Wiener filter may be used.

In addition, in a case where the filter used in the filter applicationprocess PF is fixed, the frequency characteristics of the filter may beobtained in advance. Accordingly, in a case where the filter is fixedlyused in the filter application process PF, it is possible to derive inadvance the “gain according to the target frequency characteristic ofthe image filtering process P specified based on the point spreadfunction according to the F-number when the original image data isacquired, based on the frequency characteristic of the filter and thepoint spread function which is derived in advance. The gain derived inthis way is supplied to the filtering process unit 41 (see a “gainapplication unit 49” which will be described later) by the gainspecifying unit 43 shown in FIG. 3 or the gain acquisition unit 45 shownin FIGS. 4 and 5.

A specific example of a gain calculation method will be described later,but it is preferable that the “target frequency characteristic of theimage filtering process P” or the “gain according to the targetfrequency characteristic of the image filtering process P” is obtainedin advance on a maker side of the digital camera 10 (lens unit 11 and/orthe camera main body 12) and is stored in a memory, and it is preferablethat the data stored in the memory can be appropriately used when theimage filtering process P of the original image data D1 is performed.

The target frequency characteristic of the image filtering process P isnot limited to the frequency characteristic based on the point spreadfunction (optical transfer function) of the optical system (the lens 14,the diaphragm 15, and the like), and various frequency characteristicsderived based on the imaging conditions (the above-described variousfactors of the diaphragm value (F-number) or the like) of the opticalsystem (the lens 14, the diaphragm 15, and the like) may be used as the“target frequency characteristic of the image filtering process P”.Accordingly, for example, a “frequency characteristic having differentpeaks according to the diaphragm value (F-number)” may be set as the“target frequency characteristic of the image filtering process P”.

Next, a circuit configuration example in which the image filteringprocess P (the plurality of times of filtering processes) is performedwill be described.

It is preferable that the number N of filtering processes (the firstfiltering process F1 to the N-th filtering process FN) that form theimage filtering process P is smaller than the number M of filters in acase where the filters are designed according to factors in the methodaccording to the related art. The number N of filtering processes thatform the image filtering process P is set so that N is preferably equalto or lower than 50% of M, more preferably, is equal to or lower than30% of M, and still more preferably, is equal to or lower than 10% of M.Here, an upper limit of the number N of filtering processes that formthe image filtering process P is the number of taps (maximum tap number)of a filter having the maximum tap number among filters used in theplurality of times of filtering processes. This is because the types offilters do not increase to exceed the number of taps of the filter.

The number N of filtering processes that form the image filteringprocess P is preferably equal to or smaller than 10, more preferably, isequal to or smaller than 6, and still more preferably, is equal to orsmaller than 4, but a specific numerical value example of the number Nof filtering processes is not limited thereto. Further, the number oftaps of a plurality of filters used in the plurality of times offiltering processes (filtering process unit 41) that form the imagefiltering process P may be the same between filters, or may be differentfrom each other.

In the following description relating to FIGS. 9 to 12, for ease ofdescription, a case where plurality of times of filtering processesperformed by the filtering process unit 41 are configured by the firstfiltering process F1 and the second filtering process F2 assuming that“N=2” will be described. Here, in a case where “N” is an integer whichis equal to or greater than 3 and three or more times of filteringprocesses are performed by the filtering process unit 41, it issimilarly possible to realize the filtering process unit 41 using thesame design method.

FIG. 9 is a diagram showing an example of a circuit configuration of thefiltering process unit 41. FIG. 9 shows an example of a case wherecircuits that respectively perform the first filtering process F1 to theN-th filtering process FN (here, in this example, “N=2”) are connectedin series.

The filtering process unit 41 in this example uses filtering processdata D6-1 acquired in the first filtering process F1 as processingtarget data D3-2 in the second filtering process F2. That is, thefiltering process unit 41 includes a first filter application unit 48-1,a first gain application unit 49-1, and a first process data calculationunit 50-1, and accordingly, the first filtering process F1 is performed.Further, the filtering process unit 41 includes a second filterapplication unit 48-2, a second gain application unit 49-2, and a secondprocess data calculation unit 50-2, and accordingly, the secondfiltering process F2 is performed.

The first filter application unit 48-1 applies a filter for the firstfiltering process to processing target data D3-1 of the first filteringprocess F1 to acquire filter application process data D4-1. The firstgain application unit 49-1 applies a gain g₀ for the first filteringprocess to the filter application process data D4-1 output from thefirst filter application unit 48-1 to acquire gain application processdata D5-1. Similarly, the second filter application unit 48-2 applies afilter for the second filtering process to processing target data D3-2of the second filtering process F2 to acquire filter application processdata D4-2. The second gain application unit 49-2 applies a gain g₁ forthe second filtering process to the filter application process data D4-2output from the second filter application unit 48-2 to acquire gainapplication process data D5-2.

The filter used in each filtering process may be retained by each filterapplication unit 48 (the first filter application unit 48-1 and thesecond filter application unit 48-2), or may be stored in a memory (notshown) to be appropriately read by each filter application unit 48.Further, the gain used in each filtering process may be supplied to eachgain application unit 49 (the first gain application unit 49-1 and thesecond gain application unit 49-2) by the gain specifying unit 43 shownin FIG. 3 and the gain acquisition unit 45 shown in FIG. 4 or 5.

Further, the filtering process unit 41 in this example further includesa first process data calculation unit 50-1 and a second process datacalculation unit 50-2 that add the processing target data D3 and thegain application process data D5 and output filtering process data D6.That is, the first process data calculation unit 50-1 adds theprocessing target data D3-1 to the gain application process data D5-1output from the first gain application unit 49-1 and outputs the resultto the filtering process data D6-1. Similarly, the second process datacalculation unit 50-2 adds the processing target data D3-2 to the gainapplication process data D5-2 output from the second gain applicationunit 49-2 and outputs the result to the filtering process data D6-2.

In this example in which the image filtering process P is configured bythe first filtering process F1 and the second filtering process F2, theprocessing target data D3-1 in the first filtering process F1corresponds to the original image data D1, and the filtering processdata D6-2 in the second filtering process F2 corresponds to theprocessed image data D2.

FIG. 10 is a diagram showing another example of the circuitconfiguration of the filtering process unit 41. FIG. 10 shows an exampleof a case where circuits that respectively perform the first filteringprocess F1 to the N-th filtering process FN (here, in this example,“N=2”) are connected in parallel.

In this example, the first filter application unit 48-1 and the firstgain application unit 49-1 relating to the first filtering process F1are connected in series, and the second filter application unit 48-2 andthe second gain application unit 49-2 relating to the second filteringprocess F2 are connected in series. Further, “the first filterapplication unit 48-1 and the first gain application unit 49-1 relatingto the first filtering process F1” and “the second filter applicationunit 48-2 and the second gain application unit 49-2 relating to thesecond filtering process F2” are connected in parallel. In addition, thefiltering process unit 41 includes an adder unit 52 that adds the gainapplication process data D5-1 output from the first gain applicationunit 49-1 and the gain application process data D5-2 output from thesecond gain application unit 49-2. Furthermore, the filtering processunit 41 includes a processed image data calculation unit 51 thatacquires processed image data D2 by adding up addition data D7 obtainedby adding up the gain application process data D5 (filtering processdata D6) acquired in all the filtering processes (in this example, thefirst filtering process F1 and the second filtering process F2) and theprocessing target data D3.

The filtering process unit 41 in this example having the circuitconfiguration shown in FIG. 10 uses the same data (that is, the originalimage data D1) as the processing target data D3 in the first filteringprocess F1 and the second filtering process F2. Further, the filteringprocess unit 41 acquires the processed image data D2 based on thefiltering process data D6-1 acquired in the first filtering process F1and the filtering process data D6-2 acquired in the second filteringprocess F2. In this example, the gain application process data D5 outputfrom the gain application unit 49 of each filtering process becomes thefiltering process data D6 obtained in each filtering process.

FIG. 11 is a diagram showing another example of the circuitconfiguration of the filtering process unit 41. FIG. 11 shows anotherexample in which circuits that perform each of the first filteringprocess F1 to the N-th filtering process FN (here, in this example,“N=2”) are connected in parallel.

Even in this example, similar to the example shown in FIG. 10, the firstfilter application unit 48-1 and the first gain application unit 49-1relating to the first filtering process F1 are connected in series, andthe second filter application unit 48-2 and the second gain applicationunit 49-2 relating to the second filtering process F2 are connected inseries. Further, “the first filter application unit 48-1 and the firstgain application unit 49-1 relating to the first filtering process F1”and “the second filter application unit 48-2 and the second gainapplication unit 49-2 relating to the second filtering process F2” areconnected in parallel. Accordingly, the filtering process unit 41 inthis example also uses the same data (that is, the original image dataD1) as the processing target data D3 in the first filtering process F1and the second filtering process F2.

Here, in this example, the gain application process data D5 (thefiltering process data D6) acquired in all the filtering processes (inthis example, the first filtering process F1 and the second filteringprocess F2) is added up by the processed image data calculation unit 51to acquire the processed image data D2. That is, the filtering processunit 41 adds the filtering process data D6-1 acquired in the firstfiltering process F1 and the filtering process data D6-2 acquired in thesecond filtering process F2 in the processed image data calculation unit51 to acquire the processed image data D2. In this example, the gainapplication process data D5 output from the gain application unit 49 ineach filtering process becomes the filtering process data D6 obtained ineach filtering process.

Further, it is preferable that the image filtering process P (the firstfiltering process F1 to the N-th filtering process FN) in this exampleis set so that a filter response in a case where a frequency (ω) is 0 isset to 1 (φ_(n)(0)=1) and a brightness relating to an arbitrary gain isuniformly adjusted in order to prevent change in brightness (directcurrent (DC) component) of a processing target image.

In the above-described examples shown in FIGS. 9 to 11, “the filterapplication unit 48 and the gain application unit 49” are individuallyprovided with respect to each of the plurality of times of filteringprocesses, but the plurality of times of filtering processes may beperformed by “the filter application unit 48 and the gain applicationunit 49” which are commonly provided.

FIG. 12 is a diagram showing another example of the circuitconfiguration of the filtering process unit 41. FIG. 12 shows an exampleof a configuration of a circuit that performs a plurality of times offiltering processes by “the single filter application unit 48 and thesingle gain application unit 49”. The circuit configuration exampleshown in FIG. 12 shows a function which is equivalent to the circuitconfiguration function of the above-described serial connection typeshown in FIG. 9 with respect to the filter application process PF andthe gain application process PG, but is different from the circuitconfiguration shown in FIG. 9 in that the plurality of times offiltering processes are performed by the single “filter application unit48 and gain application unit 49”.

That is, the filtering process unit 41 in this example includes thefilter application unit 48 that applies a filter to the processingtarget data D3 to acquire the filter application process data D4, andthe gain application unit 49 that applies a gain to the filterapplication process data D4 to acquire the gain application process dataD5. Further, the filtering process unit 41 includes the process datacalculation unit 50 that adds the gain application process data D5 andthe processing target data D3 to acquire the filtering process data D6,and a repetitive computation determination unit 54 that determineswhether repetition of computation processes in the filter applicationunit 48, the gain application unit 49, and the process data calculationunit 50 is necessary.

The repetitive computation determination unit 54 determines whether thenumber of times of filtering processes in the filter application unit48, the gain application unit 49, and the process data calculation unit50 reaches a predetermined number N (in this example, “N=2”). In a casewhere the number of times of filtering processes in the filterapplication unit 48, the gain application unit 49, and the process datacalculation unit 50 does not reach N, the repetitive computationdetermination unit 54 feeds back the filtering process data D6 outputfrom the process data calculation unit 50. If the filtering process dataD6 is fed back, the filter application unit 48, the gain applicationunit 49, and the process data calculation unit 50 repeat the series ofprocesses using the fed-back filtering process data D6 as new processingtarget data D3. In a case where the number of times of filteringprocesses by the filter application unit 48, the gain application unit49, and the process data calculation unit 50 reaches N, the repetitivecomputation determination unit 54 outputs the filtering process data D6which is finally output from the process data calculation unit 50 as theprocessed image data D2.

In this example, a filter h_(n) used in the filter application unit 48is supplied to the filter application unit 48 from a filter supply unit56, a gain g_(n) used in the gain application unit 49 is supplied to thegain application unit 49 from a gain supply unit 55. That is,“information about a filter h_(n-1) and a gain g_(n-1) used in an n-thfiltering process (here, “1≦n≦N”) is retained in the filter supply unit56 and the gain supply unit 55, or is retained in a memory (not shown).The filter supply unit 56 and the gain supply unit 55 receivesinformation indicating that “a filtering process performed in the filterapplication unit 48, the gain application unit 49, and the process datacalculation unit 50 is an n-th filtering process” from the repetitivecomputation determination unit 54, and supplies the filter h_(n-1) andthe gain g_(n-1) to the filter application unit 48 and the gainapplication unit 49 based on the information from the repetitivecomputation determination unit 54.

Accordingly, the filter application unit 48 shown in FIG. 12 acquiresthe filter application process data D4 using a filter h₀ for the firstfiltering process in the first filtering process F1, and acquires thefilter application process data D4 using a filter h₁ for the secondfiltering process in the second filtering process F2. Further, the gainapplication unit 49 acquires the gain application process data D5 usinga gain g₀ for the first filtering process in the first filtering processF1, and acquires the gain application process data D5 using a gain g₁for the second filtering process in the second filtering process F2.

The gain supply unit 55 that supplies a gain to the gain applicationunit 49 is configured by the gain specifying unit 43 in a case where theimage processing unit 35 includes a system configuration shown in FIG.3, and is configured by the gain acquisition unit 45 in a case where theimage processing unit 35 includes a system configuration shown in FIG. 4or FIG. 5.

The above-described circuit configurations shown in FIGS. 9 to 12 areonly examples, and the filtering process unit 41 that performs theplurality of times of filtering processes may be realized by variouscircuit configurations, and a specific circuit configuration thereof isnot particularly limited.

For example, in FIG. 12, an example of a circuit configuration in whicha filtering process (image filtering process P) which is equivalent tothe serial connection type shown in FIG. 9 by the single “filterapplication unit 48 and gain application unit 49” is shown, butsimilarly, a filtering process (image filtering process P) which isequivalent to the juxtaposition connection type shown in FIG. 10 or 11may be performed by the circuit configuration having the single “filterapplication unit 48 and gain application unit 49”.

Further, in the circuit configuration of the serial connection typeshown in FIG. 9, the first process data calculation unit 50-1 and thesecond process data calculation unit 50-2 may not be provided. That is,the gain application process data D5-1 output from the first gainapplication unit 49-1 may be used as the filtering process data D6-1 ofthe first filtering process F1 and the processing target data D3-2 ofthe second filtering process F2, and the gain application process dataD5-2 output from the second gain application unit 49-2 may be used asthe filtering process data D6-2 of the second filtering process F2 andthe processed image data D2. Similarly, in the circuit configurationshown in FIG. 12, the process data calculation unit 50 may not beprovided, and in this case, the gain application process data D5 outputfrom the gain application unit 49 may be used as the filtering processdata D6.

<Gain Determination Method>

Next, a gain determination method used in the gain application unit 49(a first gain application unit 49-1 to an N-th gain application unit49-N) will be described.

FIG. 13 is a block diagram showing an example of a functionalconfiguration of the gain acquisition unit 45, and shows the gainacquisition unit 45 which is suitably combined with the image processingunit 35 shown in FIG. 3. The gain acquisition unit 45 in this exampleincludes a reference image acquisition unit 60, a reference imageanalysis unit 61, a target frequency characteristic acquisition unit 62,and an application gain calculation unit 63.

The reference image acquisition unit 60 acquires reference image data inwhich the optical characteristic of the optical system used when theoriginal image data D1 is acquired, that is, an optical characteristicaccording to an imaging condition when the original image data D1 isacquired is reflected. It is preferable that the reference image data isacquired with respect to various lens units 11 (optical systems) thatmay be mounted on the camera main body 12 when the original image dataD1 is acquired, and that the reference image data is acquired withrespect to an imaging condition capable of being employed when theoriginal image data D1 is acquired. For example, in a case where it ispossible to acquire the original image data D1 through imaging using aplurality of types of lens units 11 (optical systems), it is preferablethat the reference image data is acquired with respect to each of theplurality of types of lens units 11 (optical systems). Further, in acase where the F-number shown in FIG. 8 is assumed as an imagingcondition, it is preferable that the reference image data is acquiredwith respect to each of all the employable F-numbers (in the exampleshown in FIG. 8, a total of 12 F-numbers from F1 to F45).

A method for acquiring the reference image data by the reference imageacquisition unit 60 is not particularly limited. For example, a methodfor performing imaging while guiding a user in a state where the lensunit 11 including the optical system which is a measurement target ismounted in a camera main body 12 to input reference image data to thereference image acquisition unit 60 from the imaging element 24 may beused. Further, in a case where reference image data acquired byperforming imaging in advance using the lens unit 11 including theoptical system which is a measurement target is stored in a memory (themain body storage unit 29 or the like shown in FIG. 1), the referenceimage acquisition unit 60 may read the reference image data relating tothe optical system which is the measurement target from the memory.

A subject for reference image data is not particularly limited, and thereference image data may be acquired by imaging a subject having aspecific shape, pattern, and color, or may be acquired by imaging asubject having an unspecified shape, pattern, and color. In a case wherereference image data is acquired by imaging a subject having a specificshape, pattern, and color, it is preferable that the reference imageacquisition unit 60 controls a display or the like on the display unit27 in cooperation with the device controller 34 (display controller 36or the like) and performs induction of imaging of the subject. Further,in a case where reference image data is acquired by imaging a subjecthaving an unspecific shape, pattern, and color, the reference imageacquisition unit 60 may analyze previously captured image data stored inthe main body storage unit 29, may select data for each imagingcondition acquired using the lens unit 11 including the optical systemwhich is the measurement target, which is data having a preferable imagecharacteristic as reference image data, from the previously capturedimage data, and may specify the result as reference image data.

The reference image analysis unit 61 performs analysis of the referenceimage data acquired by the reference image acquisition unit 60, andacquires the optical characteristic according to the imaging conditionwhen the original image data D1 is acquired. Specifically, the referenceimage analysis unit 61 may perform frequency analysis of the referenceimage data, for example, to acquire a point spread function (OTF, or thelike) for each imaging condition of the optical system which is themeasurement target.

The target frequency characteristic acquisition unit 62 specifies atarget frequency characteristic in the image filtering process P of theoriginal image data D1 which is acquired through imaging using theoptical system, based on “the optical characteristic according to theimaging condition” acquired by the reference image analysis unit 61, foreach imaging condition. For example, in a case where the image filteringprocess P is a process for canceling a point spread phenomenon of theoptical system, the target frequency characteristic acquisition unit 62may build a filter using an arbitrary design standard such as a Wienerfilter, based on a point spread function (OTF, or the like) for eachimaging condition of the optical system acquired by the reference imageanalysis unit 61, to acquire the target frequency characteristic of theimage filtering process P.

The application gain calculation unit 63 acquires a gain (candidatedata) to be applied to filter application process data in each of aplurality of times of filtering processes, based on the “targetfrequency characteristic of the image filtering process P” specified bythe target frequency characteristic acquisition unit 62. A gainacquisition method in the application gain calculation unit 63 is notparticularly limited, and for example, a method for calculating a gainusing a least squares method may be used, for example. That is, theapplication gain calculation unit 63 may fit the frequencycharacteristic of the image filtering process P using the least squaresmethod based on the frequency characteristic of each of the plurality oftimes of filtering processes, with respect to the “target frequencycharacteristic of the image filtering process P” specified by the targetfrequency characteristic acquisition unit 62, to acquire a gain to beapplied to the filter application process data in each of the pluralityof times of filtering processes.

The application gain calculation unit 63 stores the acquired gain in thegain candidate data storage unit 44 as “gain candidate data”. It ispreferable that the gain candidate data is stored in the gain candidatedata storage unit 44 in association with gain selection conditionsrelating to the target frequency characteristic of the image filteringprocess P. As the gain selection conditions, characteristic dataindicating various factors capable of affecting the target frequencycharacteristic of the image filtering process P may be suitably used,and particularly, in this example, “the imaging condition when theoriginal image data D1 is acquired” is regarded as the gain selectionconditions. Accordingly, in a case where the target frequencycharacteristic of the image filtering process P is “the frequencycharacteristic based on the point spread function (optical transferfunction) of the optical system (lens 14, diaphragm 15, and the like)used when the original image data D1 is acquired by imaging”,characteristic data relating to various factors that affect the pointspread function (for example, imaging conditions such as a diaphragmvalue (F-number), a zoom value (focal length), a subject distance, afocal position, an optical system type, a sensor SN ratio of the imagingelement 24, an image height (in-image position), or an individualoptical system difference, in imaging for acquiring the original imagedata D1) may be associated with the gain candidate data as the “gainselection conditions”.

In this way, in a case where the gain candidate data is stored in thegain candidate data storage unit 44 in association with the gainselection condition, it is preferable that the reference imageacquisition unit 60 acquires a plurality of pieces of reference imagedata which are acquired through imaging under a plurality of imagingconditions (gain selection conditions), and it is preferable that thereference image analysis unit 61 analyzes a plurality of pieces ofreference image data which are acquired through imaging under theplurality of imaging conditions to acquire optical characteristics.Further, it is preferable that the target frequency characteristicacquisition unit 62 acquires the target frequency characteristic of theimage filtering process P relating to the plurality of imagingconditions from optical characteristics under the plurality of imagingconditions, and it is preferable that the application gain calculationunit 63 acquires gains associated with the plurality of imagingconditions, and stores the gains associated with the correspondingimaging conditions (gain selection conditions) in the gain candidatedata storage unit 44.

FIG. 14 is a flowchart showing a gain acquisition flow of the gainacquisition unit 45 shown in FIG. 13. In this example, as describedabove, first, reference image data which is acquired through imagingusing the optical system which is a measurement target is acquired bythe reference image acquisition unit 60 (S11 in FIG. 14), and analysisof the reference image data is performed by the reference image analysisunit 61 to acquire an optical characteristic for each imaging conditionof the optical system which is the measurement target (S12). Further,the target frequency characteristic of the image filtering process P isacquired by the target frequency characteristic acquisition unit 62 froman optical characteristic for each imaging condition of the opticalsystem which is the measurement target is acquired (S13), and gaincandidate data for each imaging condition is acquired from the targetfrequency characteristic by the application gain calculation unit 63(S14).

The candidate data of the gain acquired in this way is stored in thegain candidate data storage unit 44 by the application gain calculationunit 63 as “gain table information” associated with the opticalcharacteristic for each imaging condition of the optical system which isthe measurement target (S15). Thus, the gain specifying unit 43 mayspecify a gain to be applied to the filter application process data ineach of the plurality of times of filtering processes from the gaincandidate data stored in the gain candidate data storage unit 44, andmay transmit the specified gain to the filtering process unit 41.

FIG. 15 is a block diagram showing another example of the functionalconfiguration of the gain acquisition unit 45, and shows the gainacquisition unit 45 which is suitably combined with the image processingunit 35 shown in FIGS. 4 and 5. The gain acquisition unit 45 in thisexample includes the target frequency characteristic acquisition unit 62and the application gain calculation unit 63, in which the targetfrequency characteristic acquisition unit 62 acquires a target frequencycharacteristic of the image filtering process P from the frequencycharacteristic storage unit 68 of the frequency analysis unit 66.

The frequency analysis unit 66 includes the reference image acquisitionunit 60, the reference image analysis unit 61, a target frequencycharacteristic specifying unit 67, and a frequency characteristicstorage unit 68. The reference image acquisition unit 60 and thereference image analysis unit 61 in this example acquire reference imagedata, similar to the above-described reference image acquisition unit 60and the reference image analysis unit 61 as shown in FIG. 13, andacquires the optical characteristic for each imaging condition of theoptical system used when the original image data D1 is acquired.Further, the target frequency characteristic specifying unit 67 in thisexample specifies a target frequency characteristic for each imagingcondition, in the image filtering process P, of the original image dataD1 which is acquired through imaging using the optical system, based onthe optical characteristic for each imaging condition of the opticalsystem acquired by the reference image analysis unit 61, similar to theabove-described target frequency characteristic acquisition unit 62shown in FIG. 13.

Here, the target frequency characteristic specifying unit 67 in thisexample stores data relating to the target frequency characteristic ofthe specified image filtering process P in the frequency characteristicstorage unit 68 in association with data on a corresponding opticalsystem and the imaging condition. Here, the “data on the correspondingoptical system and the imaging condition” may be arbitrary data capableof individually specifying an optical system which is a measurementtarget and the imaging condition, and for example, may be represented asa combination of type information (for example, a model number, or thelike) and the imaging condition (for example, F-number, or the like)allocated to the individual optical system (lens unit 11), or the like.

The target frequency characteristic acquisition unit 62 in this exampleacquires type information of an optical system (data on a correspondingoptical system) used in imaging of the original image data D1 which is atarget of the image filtering process P in the filtering process unit 41and the imaging condition of original image data D1, and acquires datarelating to a target frequency characteristic of the image filteringprocess P associated with the optical system used in imaging of theoriginal image data D1 and the imaging condition from the frequencycharacteristic storage unit 68 based on the type information and theimaging condition. The application gain calculation unit 63 acquires thegain to be applied to the filter application process data in each of theplurality of times of filtering processes based on the “target frequencycharacteristic of the image filtering process P” specified by the targetfrequency characteristic acquisition unit 62, and supplies the result tothe filtering process unit 41.

In this way, the gain acquisition unit 45 in this example reads out the“target frequency characteristic of the image filtering process P”relating to the optical system used in imaging of the original imagedata D1” and the imaging condition from the frequency characteristicstorage unit 68 to acquire the gain. Accordingly, it is possible toseparate the “process for acquiring the target frequency characteristicof the image filtering process P” in the frequency analysis unit 66 andthe “process for acquiring the gain which is actually used in thefiltering process unit 41” in the gain acquisition unit 45.

FIG. 16 is a flowchart showing a frequency characteristic acquisitionflow of the frequency analysis unit 66 shown in FIG. 15. In thisexample, as described above, first, reference image data which isacquired through imaging using the optical system which is a measurementtarget is acquired for each imaging condition by the reference imageacquisition unit 60 (S21 in FIG. 16), and analysis of the referenceimage data is performed by the reference image analysis unit 61, and anoptical characteristic for each imaging condition of the optical systemwhich is the measurement target is acquired (S22). Further, the targetfrequency characteristic of the image filtering process P is specifiedfrom the optical characteristic for each imaging condition of theoptical system which is the measurement target by the target frequencycharacteristic specifying unit 67 (S23) for each imaging condition, andis stored in the frequency characteristic storage unit 68 (S24).

FIG. 17 is a flowchart showing a gain acquisition flow of the gainacquisition unit 45 shown in FIG. 15. In this example, as describedabove, first, data relating to a target frequency characteristic of theimage filtering process P associated with an optical system and theimaging condition used in imaging of the original image data D1 which isa target of the image filtering process P in the filtering process unit41 is acquired from the frequency characteristic storage unit 68 by thetarget frequency characteristic acquisition unit 62 (S31 in FIG. 17).Further, a gain to be used in the filtering process is acquired from thetarget frequency characteristic by the application gain calculation unit63 (S32), and is supplied to the filtering process unit 41 (S33).

<Gain Coefficient (Gain Value Vector) Calculation Method>

Hereinafter, a specific flow of a computation process in theabove-described gain acquisition unit 45 (particularly, the applicationgain calculation unit 63) will be described. An example, a case in whichthe filtering process unit 41 has the circuit configuration shown inFIG. 11 will be mainly described, but in a case where the filteringprocess unit 41 has another circuit configuration (see FIGS. 9, 10, and12), it is similarly possible to calculate a gain by the samecomputation process.

A method described below employs a least squares standard in whichweighting is performed based on each of a frequency and an in-imageposition (pixel position), and calculates a gain so that a frequencycharacteristic of the entirety of the image filtering process Pperformed by the filtering process unit 41 is close to a desired targetfrequency characteristic.

In the circuit configuration shown in FIG. 11, when a frequencycharacteristic of a filter used in a filter application unit 48-n thatperforms an n-th filtering process Fn (here, “1≦n≦N”) is represented as“φ_(n-1)(ω)” and a gain used in a gain application unit 49-n isrepresented as “g_(n-1)”, the frequency characteristic of the n-thfiltering process Fn is represented as “g_(n-1)φ_(n-1)(ω)”. Accordingly,when the frequency characteristic of the entirety of the image filteringprocess P (first filtering process F1 to N-th filtering process FN)performed by the filtering process unit 41 is represented as “f(ω)”,“f(ω)=g₀φ₀(ω) . . . +g_(N-1)φ_(N-1)(ω) (here, “N” is an integer which isequal to or greater than 2) is established.

In the circuit configuration shown in FIGS. 9 and 12, when the frequencycharacteristic of the entirety of the n-th filtering process Fn isrepresented as “1+g_(n-1)(ψ_(n-1)(ω)−1)”, the frequency characteristicφ_(n-1)(ω) of the filter used in each filter application unit 48 isrepresented as “φ_(n-1)(ω)=ψ_(n-1)(ω)−1”, and the frequencycharacteristic f(ω) of the entirety of the image filtering process P isrepresented as “f(ω)=(1+g₀φ₀(ω)) . . . ×(1+g_(N-1)φ_(N-1)(ω)). Further,in the circuit configuration shown in FIG. 10, the frequencycharacteristic f(ω) of the entirety of the image filtering process P isrepresented as “f(ω)=1±g₀φ₀(ω) . . . +g_(N-1)φ_(N-1)(ω).

Accordingly, the following description relating to the circuitconfiguration shown in FIG. 11 may be applied to the circuitconfiguration shown in FIG. 10 by applying the frequency characteristic“f(ω)=g₀φ₀(ω) . . . +g_(N-1)φ_(N-1)(ω)” of the circuit configurationshown in FIG. 11 to the frequency characteristic “f(φ)=1+g₀φ₀(ω) . . .+g_(N-1)φ_(N-1)(ω). For example, by considering “1+g₀φ₀(ω)” in thefrequency characteristic f(ω) of the circuit configuration shown in FIG.10 as “g₀φ₀(ω)” in the frequency characteristic f(ω) of the circuitconfiguration shown in FIG. 11, the following description relating tothe circuit configuration shown in FIG. 11 may be applied to the circuitconfiguration shown in FIG. 10. Similarly, the following descriptionrelating to the circuit configuration shown in FIG. 11 may be applied tothe circuit configuration shown in FIGS. 9 and 12 by applying thefrequency characteristic “f(ω)=g₀φ₀(ω) . . . +g_(N-1)φ_(N-1)(ω)” of thecircuit configuration shown in FIG. 11 to “f(ω)=(1+g₀φ₀(ω)) . . .×(1+g_(N-1)φ_(N-1)(ω))”. In this case, a product computing expression ofthe frequency characteristic f(ω) of the circuit configuration shown inFIGS. 9 and 12 is converted into a sum computing expression usinglogarithmic processing, and the following description relating to thecircuit configuration of FIG. 11 is easily applied to the circuitconfiguration shown in FIGS. 9 and 12.

Here, it is assumed that the filter used in the filter application unit48-n of the n-th filtering process Fn is a filter for image restorationbased on the point spread function of the optical system (the lens 14,the diaphragm 15, and the like) used when the original image data D1 isacquired through imaging. As described later, since the filter for imagerestoration based on the point spread function has a two-dimensionaldegree of freedom with respect to the frequency ω and has atwo-dimensional degree of freedom with respect to an in-image position(pixel position) r, a frequency characteristic having total “fourdegrees of freedom” is obtained.ωεR ²R²: two-dimensional real number column vector space

If the frequency vector ω and the in-image position vector r are used,the frequency characteristic of the filter used in the filterapplication unit 48-n of the n-th filtering process Fn may berepresented as “φ_(n-1)(ω, r)”. By performing discretization of thefrequency characteristic φ_(n)(ω, r) of the filter with respect to N_(ω)sampling points relating to the frequency and N_(r) sampling pointsrelating to the in-image position, the following vector φ_(n) isobtained.

ϕ_(n) ∈ C^(N_(ω)N_(r))$\phi_{n} = {\sum\limits_{k = 1}^{N_{r}}{e_{k}^{\prime} \otimes \left( {\sum\limits_{j = 1}^{N_{\omega}}{e_{j} \otimes {\phi_{n}\left( {\omega_{j},r_{k}} \right)}}} \right)}}$C^(N): N-dimensional complex column vector space

: Kronecker producte_(j): standard base of N_(ω)-dimensional column vector spacee′_(k): standard base of N_(r)-dimensional column vector space

The frequency “ω_(j)” and the in-image position “r_(k)” may berepresented as the following expressions, in which “ω_(j)” representsN_(ω) sampling points relating to the frequency, and “r_(k)” representsN_(r) sampling points relating to the in-image position.{ω_(j)}_(j=1) ^(N) ^(ω){r _(k)}_(k=1) ^(N) ^(r)

The gain vector (gain group) g configured by gains used in the pluralityof times of filtering processes (the first filtering process F1 to theN-th filtering process FN) is represented as the following expression.g=[g ₀ g ₁ . . . g _(N-1)]^(T)T: transpose

The frequency characteristic “f(ω, r|g)” of the entirety (entirety ofthe filtering process unit 41) of the image filtering process P in whichthe gains represented as the above expression are set is represented asthe following expression in the circuit configuration shown in FIG. 11.

${f\left( {\omega,{r❘g}} \right)} = {\sum\limits_{i = 0}^{N - 1}{g_{i}{\phi_{i}\left( {\omega,r} \right)}}}$

On the other hand, a “target frequency characteristic “d(ω, r)” of theimage filtering process P″ to be realized may be determined by anarbitrary method, and the determination method is not particularlylimited. Generally, a restoration filter based on a point spreadfunction may be suitably realized by a convolution-type filter. As amethod for determining a frequency characteristic of the filter based onthe point spread function, various methods may be used, and for example,a Wiener filter is widely used in consideration of an SN ratio of animaging optical system. In the Wiener filter, a frequency characteristicof the filter may be represented based on an optical transfer function(OTF) of the point spread function and information about the SN ratio,like the following expression. Accordingly, the target frequencycharacteristic “d(ω, r)” of the image filtering process P to be realizedmay be determined based on the Wiener filter.

${d\left( {\omega,r} \right)} = \frac{H^{*}\left( {\omega,r} \right)}{{{H\left( {\omega,r} \right)}}^{2} + {1/{{SNR}\left( {\omega,r} \right)}}}$H(ω, r): OTF of optical systemH*(ω, r): complex conjugate of OTF of optical systemSNR(ω, r): SN ratio of imaging system

The frequency characteristic of the entirety of the image filteringprocess P is represented as “f(ω, r|g)”, the target frequencycharacteristic of the image filtering process P to be realized isrepresented as “d(ω, r)”, and an approximation weight function isrepresented as “w(ω, r)”. In this case, a gain vector that minimizes ageneric function “J_(LMS)[g]” based on a weighted least squares standardrepresented as the following expression may be used as a gain vectorindicating an optimal frequency characteristic in the image filteringprocess P.J _(LMS) [g]=∫∫w(ω,r)∥f(ω,r|g)−d(ω,r)∥² dωdr

If the above expression is discretized by sampling points of thefrequency and the in-image position, the following expression isobtained.J _(LMS) [g]=∥W ^(1/2)(Ag−d)∥²

where

W=diag[w]

A=[φ₀ φ₁ . . . φ_(N-1)]

In the above expression, “diag” represents a diagonal matrix in which anargument vector is a diagonal element, “w” and “d” in the aboveexpression are obtained by respectively vectoring “w(ω, r)” and “d(ω,r), in a similar way to the above-described method relating to “vectorφ_(n)”.

An optimal solution g_(OPT) of the gain vector is represented as thefollowing expression, based on the above expression.

$g_{OPT} = {{\arg\underset{g}{\min\;}{J_{LMS}^{\prime}\lbrack g\rbrack}} = {\left( {A^{H}{WA}} \right)^{- 1}A^{H}{Wd}}}$H: Hermitian Transpose

If the approximation weight function and the frequency characteristic(filter coefficient) of each filter are determined, a portion of“(A^(H)WA)⁻¹A^(H)W” in the above expression is represented as a matrixcapable of being calculated in advance. The optimal solution g_(OPT) ofthe gain vector may be calculated by a computation for applying a matrixto a filter characteristic acquired based on a point spread functionindicating an individual image deterioration characteristic.

In a case where the filtering process unit 41 employs the circuitconfiguration shown in FIG. 11, in order to prevent change in a DCcomponent (brightness) of an image, it is preferable that a constraintcondition that an amplification factor (gain) of the DC component is setto 1.0 times is applied, in addition to the above-described condition.Specifically, acquisition of the optimal solution g_(OPT) of the gainvector according to the following expression corresponds to “the settingof the amplification factor of the DC component to 1.0 time in order toprevent change in the brightness of the entire image”.

ϕ_(i)(0, r) = ϕ_(i)(0)  0 ≤ i < N, ∀r$g_{OPT} = {\arg{\min\limits_{g}\;{J_{LMS}^{\prime}\lbrack g\rbrack}}}$${{subject}\mspace{14mu}{to}\text{:}\mspace{14mu}{\sum\limits_{i = 0}^{N - 1}{g_{i} \times {\phi_{i}(0)}}}} = 1$

The above expression may be considered as a quadratic programming (QP)problem, and may be solved by a small amount of calculation when “N” issmall. Further, as an example, by setting a limit to an amplificationfactor of a DC component of each filter as represented as the followingexpression, it is possible to exclude the limit (constraint condition)to the amplification factor of the DC component.φ₀(0)=1,φ₁(0)=0(1≦i<N)

<Review of Weighting>

In the above-described least squares method, the approximation accuracyof the frequency characteristic is given a weight based on the frequencyand the in-image position (pixel position) represented as “w(ω, r)”. Forexample, in a case where the approximation of the frequencycharacteristic based on the least squares method is performed based onthe approximation error evaluation function J_(LMS)[g], theapproximation error evaluation function J_(LMS)[g] is given a weightbased on the frequency and the in-image position (pixel position). Byadjusting the weight, it is possible to control image quality. That is,when approximating to the target frequency characteristic of the imagefiltering process P using the least squares method, by adjusting aweight with respect to a specific frequency or a specific pixelposition, it is possible to enhance approximation accuracy relating tothe specific frequency or the specific pixel position in the leastsquares method, and to control image quality.

For example, in the above-described least squares method, a weight in alow-frequency band may be set to be larger than a weight in ahigh-frequency band. Generally, since a low-frequency component iseasily perceived compared with a high-frequency component in view ofhuman visual properties, it is possible to process the low-frequencycomponent with high accuracy by prioritizing the low-frequency componentwith respect to the high-frequency component as the accuracy of the“approximation to the target frequency characteristic of the imagefiltering process P”.

Here, “the low-frequency band (low-frequency component)” and “thehigh-frequency band (high-frequency component)” represent relativefrequency ranges (frequency bands). For example, a range where asampling frequency is equal to or smaller than ¼ (=0.25 fs) may beconsidered as “the low-frequency band (low-frequency component)”, and arange where the sampling frequency is larger than 0.25 fs and is equalto or smaller than 0.5 fs may be considered as “the high-frequency band(high-frequency component)”.

Further, in the above-described least squares method, a weight in thehigh-frequency band may be set to be larger than a weight in thelow-frequency band according to an imaging condition when the originalimage data D1 is acquired. Generally, if a high-frequency characteristicof a filter is bad in a case where noise is large, the noise isamplified. Accordingly, in the image filtering process P of the originalimage data D1 acquired under an imaging condition that it is predictedthat noise is large, it may be preferable to prioritize thehigh-frequency component with respect to the low-frequency component asthe accuracy of “the approximation to the target frequencycharacteristic of the image filtering process P”.

Here, “the imaging conditions when the original image data D1 isacquired” are not particularly limited. Arbitrary factors capable ofaffecting the amount of noise may be used as imaging conditions, and forexample, a diaphragm value (F-number) and an imaging sensitivity of theimaging element 24 may be used as “the imaging condition when theoriginal image data D1 is acquired” capable of affecting the amount ofnoise.

Further, the above-described weighting in the least squares method maybe determined according to pixel positions in the original image dataD1. For example, in a case where the original image data D1 is acquiredthrough imaging using a normal digital camera used by a general user,since a main subject is disposed at a central portion of an image inmany cases, there is a tendency that high-frequency components of pixelpositions at the central portion of the image are is emphasized. On theother hand, in a case where the importance of an image in the vicinityof a boundary of an imaging range is relatively high in a monitoringcamera or the like, there is a tendency that high-frequency componentsof pixel positions in a peripheral portion of an image is emphasized. Inthis way, there is a case where the importance is changed according topixel positions in the original image data D1, and in this case, pixelpositions with high weights (prioritized pixel positions) may bedetermined according to the type of an imaging device or a usage(product) of a captured image.

For example, in an optical system provided in the digital camera 10 usedby a general user, there is a tendency that a central portion generallyhas a high resolution and thus has an excellent resolutioncharacteristic regardless of frequencies and resolution performance in aperipheral portion becomes low. Accordingly, in the image filteringprocess P of the original image data D1 acquired by imaging using thedigital camera 10 shown in FIG. 1, by increasing a weight in ahigh-frequency band at a central portion of an image and increasing aweight in a low-frequency band at a peripheral portion of the image, itis possible to enhance image quality according to opticalcharacteristics. That is, in the above-described least squares method,at pixel positions (pixel positions with a low image height) which areequal to or shorter than a first distance from the center of an image inthe original image data D1, weights in a high-frequency band may be setto be larger than those at pixel positions (pixel positions with a highimage height) which is longer than the first distance from the center ofthe image in the original image data D1. Further, in the above-describedleast squares method, at pixel positions (pixel positions with a highimage height) which is longer than a second distance from the center ofthe image in the original image data D1, weights in a low-frequency bandmay be set to be larger than those at pixel positions (pixel positionswith a low image height) which is equal to or shorter than the seconddistance from the center of the image in the original image data D1.

Here, it is preferable that “the high-frequency band” is included in arange which is larger than 0.25 fs and is equal to or smaller than 0.5fs, and it is preferable that “the low-frequency band” is included in arange which is equal to or smaller than 0.25 fs. Further, “the firstdistance” and “the second distance” may be the same or may be differentfrom each other.

<Review in a Case where Filter Frequency Characteristic does not IncludeFrequency Attenuation>

In a case where a plurality of times of filtering processes is performedby the filtering process unit 41 having the circuit configuration of theserial connection type shown in FIG. 9, it is preferable that a controlmethod is devised according to whether a filter frequency characteristicapplied to the processing target data D3 has frequency attenuation ineach filter application unit 48. The frequency attenuation actseffectively particularly in a case where “the level of spread (blurring)of a point image represented by a point spread function of an opticalsystem is large” and “an SN ratio is low”, and shows an effect ofreducing a noise component in a high-frequency band.

For example, in a case where it is not necessary to give an effect offrequency attenuation to a filter used in each filter application unit48, a frequency characteristic φ_(n)(ω) of the filter used in eachfilter application unit 48 of the filtering process unit 41 shown inFIG. 11 may be set based on the following expression.

${\phi_{n}(\omega)} = \left\{ \begin{matrix}{\phi_{n}(\omega)} & \left( {{{\phi_{n}(\omega)}} > 1} \right) \\1 & {otherwise}\end{matrix} \right.$

The above expression means that a filter having the original frequencycharacteristic φ_(n)(ω) is used in a case where the frequencycharacteristic φ_(n)(ω) of the filter used in each filter applicationunit 48 satisfies “|φ_(n)(ω)|>1” and a filter in which “frequencycharacteristic (response) is 1” is used in a case where “|φ_(n)(ω)|≦1”is satisfied.

In a case where it is not necessary to give the effect of frequencyattenuation to the filter used in each filter application unit 48, thefrequency characteristic φ_(n)(ω) of the filter used in each filterapplication unit 48 of the filtering process unit 41 shown in FIGS. 9,10, and 12 may be set based on the following expression, for example.

${\phi_{n}(\omega)} = \left\{ \begin{matrix}{\phi_{n}(\omega)} & \left( {{{\varphi_{n}}\left( {= {{1 + {\phi_{n}(\omega)}}}} \right)} > 1} \right) \\0 & {otherwise}\end{matrix} \right.$

Further, in a case where the circuit configuration that forms thefiltering process unit 41 is the serial connection type shown in FIG. 9,a frequency characteristic of the entirety of the image filteringprocess P is represented based on the product of frequencycharacteristics of filters used in each filter application unit 48. Inorder to similarly handle “the frequency characteristic of the entiretyof the image filtering process P of the serial connection type” and “thefrequency characteristic of the entirety of the image filtering processP of the juxtaposition connection type”, it is preferable to convert amatter of “the product” into a matter of “the sum” by performinglogarithmic processing with respect to the frequency characteristics ofall the filters used in the image filtering process P of the serialconnection type. In this case, using the same arithmetic process as thecalculation method of the optimal solution g_(OPT) of the gain vectorrelating to the filtering process unit 41 having the circuitconfiguration of the juxtaposition connection type, it is possible tocalculate the optimal solution g_(OPT) of the gain vector relating tothe filtering process unit 41 having the circuit configuration of theserial connection type. In a case where the frequency characteristic ofthe filter does not include the attenuation characteristic, thefrequency characteristic of the filter does not becomes a large negativevalue by the above-described logarithmic-processing, and logarithmicprocessing of a complex number may be performed using the followingexpression.log z=log|z|+j(∠z+2nπ)z: complex number (=x+jy=r_(c)e^(jθ))j: imaginary unitr_(c): absolute value (=√(x²+y²))x=r_(c) cos θy=r_(c) sin θθ: argument (=∠z)

<Review in a Case where Frequency Characteristic of Filter IncludesFrequency Attenuation>

For example, in the filtering process unit 41 having the circuitconfiguration of the serial connection type as shown in FIG. 9, afrequency characteristic of a filter used in each filter applicationunit 48 includes frequency attenuation, it is preferable to note thefollowing points. That is, in a case where a frequency amplificationfactor is close to 0, a value after logarithmic processing becomes alarge negative value, and thus, in a case where fitting of the frequencycharacteristic is performed using a least squares standard after thelogarithmic processing is performed, for example, there is a concernthat the large negative value greatly affects a frequency characteristicapproximation standard. Thus, in the logarithmic processing, it ispreferable to perform a countermeasure such as clipping using apredetermined negative minimum value or adjustment of reducing a weightin a frequency band where frequency attenuation easily occurs.

For example, in the case of clipping, it is preferable that thefiltering process unit 41 uses a filter that makes the filtering processdata D6 equal to the filter application process data D4 in each of theplurality of times of filtering processes at a frequency where the ratioof the processed image data D2 to the original image data D1 in a targetfrequency characteristic of the image filtering process P is smallerthan 1.

By determining and adjusting a gain used in each filtering process asdescribed above, for example, it is possible to greatly reduce thenumber of parameters to be retained, compared with a case where a filter(filter coefficient) is retained based on a point spread function withrespect to all combinations of imaging conditions, for example. Further,in a general method, if an image deterioration characteristic such as apoint spread function is not known, it is difficult to appropriatelyperform filter design (filter coefficient design), but according to thisembodiment, even before perceiving respective image deteriorationcharacteristics, it is possible to complete design of an FIR filter of afinite tap length for which a large amount of calculation is necessary.In this case, it is possible to calculate an optimal gain vector by asimple calculation after a specific image deterioration characteristicis acquired.

<Filter Design Method>

Next, a specific example of a filter design method used in each filterapplication unit 48 of the filtering process unit 41 will be described.

FIG. 18 is a block diagram showing an example of a functionalconfiguration of a filter acquisition apparatus 76.

The filter acquisition apparatus 76 that performs design and acquisitionof a filter used in each filter application unit 48 includes a filtercalculation unit 77. The filter calculation unit 77 in this exampleacquires frequency characteristic data of a plurality of types of imagefiltering processes (a first image filtering process P1 to a k-th imagefiltering process Pk (here, “k” is an integer which is equal to orgreater than 2)) specified based on optical characteristics of opticalsystems according to a plurality of imaging conditions, and outputs aplurality of filters including a reference filter h_(b) and a variancefilter h_(v). That is, the filter calculation unit 77 acquires thereference filter h_(b) determined according to an average of frequencycharacteristics of the plurality of types of image filtering processesP1 to Pk, based on the frequency characteristics of the plurality oftypes of image filtering processes P1 to Pk specified based on theoptical characteristics of the plurality of optical systems according toa plurality of imaging conditions. Further, the filter calculation unit77 acquires at least one variance filter determined according to avariance of the frequency characteristics of the plurality of types ofimage filtering processes P1 to Pk, based on the frequencycharacteristics of the plurality of types of image filtering processesP1 to Pk.

FIG. 19 is a block diagram showing an example of a functionalconfiguration of the filter calculation unit 77.

The filter calculation unit 77 in this example includes an averagecalculation unit 78, a variance calculation unit 79, and a filtercharacteristic acquisition unit 80. The average calculation unit 78calculates the average of the frequency characteristics of the pluralityof types of image filtering processes P1 to Pk from the frequencycharacteristics data of the plurality of types of image filteringprocesses P1 to Pk. The variance calculation unit 79 calculates avariance of frequency characteristics of the plurality of types of imagefiltering processes P1 to Pk, from the frequency characteristics data ofthe plurality of types off image filtering processes P1 to Pk. Thefilter characteristic acquisition unit 80 calculates the referencefilter h_(b) based on the average of the frequency characteristics ofthe plurality of types of image filtering processes P1 to Pk calculatedin the average calculation unit 78. Further, the filter characteristicacquisition unit 80 calculates the variance filter h_(v) based on theaverage of the frequency characteristics of the plurality of types ofimage filtering processes P1 to Pk calculated in the average calculationunit 78 and the variance of the frequency characteristic of theplurality of types of image filtering processes P1 to Pk calculated inthe variance calculation unit 79.

The frequency characteristic data of the plurality of types of imagefiltering processes P1 to Pk used in the filter calculation unit 77 (theaverage calculation unit 78, the variance calculation unit 79, and thefilter characteristic acquisition unit 80) in this example is specifiedbased on optical characteristics of an optical system according to aplurality of imaging conditions. In this way, by determining the“frequency characteristic data of the plurality of types of imagefiltering processes P1 to Pk” which are bases of calculation of thefilters (the reference filter h_(b) and the variance filter h_(v)) withreference to the plurality of imaging conditions, it is possible tocalculate the filters (the reference filter h_(b) and the variancefilter h_(v)) suitable for such a plurality of imaging conditions.Accordingly, by assuming “representative imaging conditions” when theoriginal image data D1 is acquired as the “plurality of imagingconditions”, it is possible to realize the “target frequencycharacteristic of the image filtering process” with high accuracy by thefiltering process unit 41.

<Frequency Characteristic of Plurality of Types of Image FilteringProcesses>

Here, a frequency characteristic of each of “the plurality of types ofimage filtering processes P1 to Pk” forms “the target frequencycharacteristic of the image filtering process P”, and in this example,is determined by a frequency characteristic based on an opticalcharacteristic of an optical system for each imaging condition capableof being employed in imaging.

For example, in a case where “the frequency characteristics of theplurality of types of image filtering processes P1 to Pk” are frequencycharacteristics based on a point spread function of an optical system,it is possible to determine “the frequency characteristics of theplurality of types of image filtering processes P1 to Pk” according tofrequency characteristics of a plurality of types of point spreadfunctions obtained by changing imaging conditions (for example,F-numbers, or the like) that may affect the point spread functions. Thatis, it is possible to determine “the frequency characteristics of theplurality of types of image filtering processes P1 to Pk” according tofrequency characteristics of a plurality of types of point spreadfunctions under different imaging conditions. “The plurality of types ofpoint spread functions” may be acquired by an arbitrary method. Forexample, the plurality of types of point spread functions may beacquired based on actual measurement values, or may be acquired based onestimation values. “The frequency characteristics of the plurality oftypes of image filtering processes P1 to Pk” may be calculated based onan arbitrary design standard such as a Wiener filter using the pluralityof types of acquired point spread functions.

“The frequency characteristics of the plurality of types of imagefiltering processes P1 to Pk” may be acquired by an arbitrary device.For example, the frequency characteristics of the plurality of types ofimage filtering processes P1 to Pk may be acquired by the filteracquisition apparatus 76, may be acquired by another device, or may beread and acquired from a memory (not shown) by the filter calculationunit 77 of the filter acquisition apparatus 76.

<Variation in Frequency Characteristics of Plurality of Types of ImageFiltering Processes>

Variation in frequency characteristics of a plurality of types of imagefiltering processes may be represented based on a unimodal distribution,or may be represented based on a multimodal distribution.

FIG. 20 is a conceptual diagram showing variation in frequencycharacteristics of a plurality of types of image filtering processes asa unimodal distribution. FIG. 21 is a conceptual diagram showingvariation in frequency characteristics of a plurality of types of imagefiltering processes as a multimodal distribution.

A variation representation method based on the unimodal distributionshown in FIG. 20 is a method for representing variation in frequencycharacteristics of a plurality of types of image filtering processes byone multi-dimensional complex normal distribution (see “variationdistribution G” in FIG. 20). On the other hand, a representation methodbased on the multimodal distribution is a method for representingvariation in frequency characteristics of a plurality of types of imagefiltering processes by a plurality of multi-dimensional complex normaldistributions (see “a first variation distribution G1” and “a secondvariation distribution G2” in FIG. 21).

FIG. 20 shows a case where a variation distribution G of frequencycharacteristics of a plurality of types of image filtering processes isrepresented by a frequency characteristic φ₀ of a filter of a firstfiltering process F1, a frequency characteristic φ₁ of a filter of asecond filtering process F2, and a frequency characteristic φ₂ of afilter of a third filtering process F3. Further, FIG. 21 shows a casewhere a first variation distribution G1 and a second variationdistribution G2 that form a variation distribution G of frequencycharacteristics of a plurality of types of image filtering processes arerespectively represented by a frequency characteristic (φ₀ ⁽¹⁾, φ₀ ⁽²⁾)of a filter of a first filtering process F1, a frequency characteristic(φ₁ ⁽¹⁾, φ₁ ⁽²⁾) of a filter of a second filtering process F2, and afrequency characteristic (φ₂ ⁽¹⁾, φ₂ ⁽²⁾) of a filter of a thirdfiltering process F3. FIGS. 20 and 21 show the frequency characteristicsφ₀ to φ₂ of the filters under the premise that the filtering processunit 41 has the circuit configuration shown in FIG. 11 as an example.

In a case where variation in frequency characteristics of a plurality oftypes of image filtering processes has the variation distribution G ofan oval shape as shown in FIG. 20, for example, in order to realize therespective image filtering processes by a plurality of filteringprocesses (the first filtering process F1 to the third filtering processF3 in the example shown in FIG. 20) with high accuracy, it is preferableto determine the frequency characteristics φ₀, φ₁, and φ₂ of therespective filtering processes based on the center, the long axis, andthe short axis of the variation distribution G. Similarly, in a casewhere variation of frequency characteristics of a plurality of types ofimage filtering processes has the first variation distribution G1 andthe second variation distribution G2 which are formed in oval shapes asshown in FIG. 21, for example, in order to represent the image filteringprocesses included in the first variation distribution G1 and the secondvariation distribution G2 by a plurality of filtering processes (in theexample shown in FIG. 21, the first filtering process F1 to the thirdfiltering process F3) with high accuracy, it is preferable to determinethe frequency characteristics φ₀ ⁽¹⁾, φ₁ ⁽¹⁾, φ₂ ⁽¹⁾, φ₀ ⁽²⁾, φ₁ ⁽²⁾ andφ₂ ⁽²⁾ in the respective filtering processes based on the center, thelong axis, and the short axis of each of the respective first variationdistribution G1 and second variation distribution G2.

<Unimodal Distribution>

Next, an example of a method for designing a filter and a gain in a casewhere variation of frequency characteristics of a plurality of types ofimage filtering processes is considered as a unimodal distribution (seeFIG. 20) will be described.

The following method relates to a method for obtaining, in a case wherea variation sample of frequency characteristics of a plurality of typesof image filtering processes of an arbitrary number is actuallyacquired, frequency characteristics of a desired number of base filters(FIR filters) capable of compensating variation of the frequencycharacteristics of the image filtering processes with high accuracy.Further, a case where modeling is performed using a unimodaldistribution as a multi-dimensional complex normal distribution toperform estimation of a maximum likelihood (ML) will be described.

A variation distribution G of a unimodal distribution can be representedbased on a multi-dimensional complex normal distribution, and themulti-dimensional complex normal distribution is characterized by acovariance matrix, and is obtained by the following expression.

$\psi_{o} = {\frac{1}{N_{p}}{\sum\limits_{i = 1}^{N_{p}}\beta_{i}}}$$R_{0} = {\frac{1}{N_{p}}{\sum\limits_{i = 1}^{N_{p}}{\left( {\beta_{i} - \psi_{o}} \right)\left( {\beta_{i} - \psi_{o}} \right)^{H}}}}$

In the above expression, “β_(i)” represents a vector obtained bydiscretizing a frequency characteristic β(ω, r) which is individuallyacquired with respect to each of a plurality of types of image filteringprocesses based on a frequency (ω) and a position (r), and “N_(p)”represents the number of the plurality of types of image filteringprocesses. Accordingly, “ψ₀” in the above expression represents anaverage (average matrix) of frequency characteristics of the pluralityof types of image filtering processes, and “R₀” in the above expressionrepresents a variance (covariance matrix) of the frequencycharacteristics of the plurality of types of image filtering processes.

By performing characteristic value decomposition of the covariancematrix represented as the above expression according to the followingexpression, it is possible to obtain a characteristic vector ψ_(i)(here, “i” is an integer which is equal to or greater than 1 and isequal to or smaller than N_(P)).

${R_{0} = {\sum\limits_{i = 1}^{N_{p}}{\lambda_{i}\psi_{i}\psi_{i}^{H}}}},\left( {\lambda_{1} \geq \lambda_{2} \geq \ldots \geq \lambda_{N_{p}}} \right)${ψ_(i)}_(i = 1)^(N_(p))

According to Karhunen-Loeve expansion, in order to averagely approximatea variation distribution by an m-dimensional linear partial space, it ismost preferable to use a partial space spread by m correspondingcharacteristic vectors ψ_(i) (here, “i” is an integer which is equal orgreater than 1 and is equal to or smaller than m) in a descending orderof characteristic values.

Accordingly, optimal frequency characteristics φ₀ to φ_(N-1) of filtersfor filtering processes for reproducing the respective frequencycharacteristics of the plurality of types of image filtering processeshaving the unimodal variation distribution G with high accuracy can beacquired based on the following expression.

ϕ₀ = ψ₀ ϕ₁ = ψ₁ ⋮ ϕ_(N − 1) = ψ_(N − 1)

As described above, since the frequency characteristic φ₀ of the filterof the first filtering process F1 corresponds to the center of thedistribution of the frequency characteristics of the plurality of typesof image filtering processes, it is preferable that a gain in the firstfiltering process F1 is fixed to “1.0” from a viewpoint of fixing thecenter of the distribution to stabilize the process. With respect togains in the other filtering processes, it is possible to calculateoptimal values by the above-described weighted least squares standard,for example.

FIG. 22 is a diagram schematically showing an example of frequencycharacteristics of a plurality of types of image filtering processescapable of being classified as a unimodal distribution, in which alateral axis represents a frequency (ω) and a longitudinal axisrepresents a response. Here, the response refers to the ratio offrequency component amounts before and after each image filteringprocess (that is, frequency component amount after process/frequencycomponent amount of data before process), and “response=1” means thatdata before and after the image filtering process is the same.

FIG. 22 shows frequency characteristics of four types of image filteringprocesses (a first image filtering process P1, a second image filteringprocess P2, a third image filtering process P3, and a fourth imagefiltering process P4). The four types of image filtering processes P1 toP4 have different frequency characteristics, but in any case, a responsein a case where a frequency (ω) is “0” shows “1”.

FIG. 23 is a diagram schematically showing an example of a variance ofthe frequency characteristics of the plurality of types of imagefiltering processes shown in FIG. 22, in which a lateral axis representsa frequency (ω) and a longitudinal axis represents a variance. FIG. 23does not strictly show the variance of the frequency characteristics ofthe plurality of types of image filtering processes P1 to P4 shown inFIG. 22, for ease of description.

Since the response in a case where the frequency (ω) is “0” shows “1” inany case of the four types of image filtering processes P1 to P4 shownin FIG. 22, the variance in a case where the frequency (ω) is “0”becomes “0”. In the example shown in FIG. 23, with respect to thefrequency characteristics of the plurality of types of image filteringprocesses P1 to P4, “a component indicating a largest variation (seereference numeral “H1” in FIG. 23)” and “a component indicating a secondlargest variation (see reference numeral “H2” in FIG. 23)” are shown.

FIGS. 24A to 24C are diagrams schematically showing examples of basefilters (characteristic vectors) acquired from an average and a varianceof the frequency characteristics of the plurality of types of imagefiltering processes shown in FIG. 22, in which FIG. 24A shows “φ₀”, FIG.24B shows “φ₁”, and FIG. 24C shows “φ₂”.

As described above, it is preferable that the frequency characteristicφ₀ of the filter used in the first filtering process F1 is determinedbased on the center (average) of the distribution of the frequencycharacteristics of the plurality of types of image filtering processesP1 to P4.

Further, it is preferable that the frequency characteristics of thefilters used in the other filtering processes P1 to P4 are determinedsequentially from a large component in variation (variance) of thefrequency characteristics of the plurality of types of image filteringprocesses shown in FIG. 23. Accordingly, it is preferable that thefrequency characteristic φ₁ of the filter used in the second filteringprocess F2 is determined based on the largest component (see “H1” inFIG. 23) in the variation of the frequency characteristics of theplurality of types of image filtering processes P1 to P4. Similarly, itis preferable that the frequency characteristic φ₂ of the filter used inthe third filtering process F3 is determined based on the second largestcomponent (see “H2” in FIG. 23) in the variation of the frequencycharacteristics of the plurality of types of image filtering processes.

The above-described example relates to a case where the image filteringprocess P includes the first filtering process F1 to the third filteringprocess F3, but the image filtering process P may be configured by twofiltering processes, or may be configured by four or more filteringprocesses. For example, in a case where the image filtering process P isconfigured by the first filtering process F1 and the second filteringprocess F2, the frequency characteristic of the filter used in the firstfiltering process F1 may be determined based on the above-described “φ₀”(see FIG. 24A), and the frequency characteristic of the filter used inthe second filtering process F2 may be determined based on theabove-described “φ₁” (see FIG. 24B).

FIGS. 25A and 25B show a frequency characteristic f(ω) (see FIG. 25A)and a gain example (see FIG. 25B) relating to the image filteringprocess P (first filtering process F1 to third filtering process F3)performed by the filtering process unit 41 having the circuitconfiguration shown in FIG. 11.

As described above, it is preferable that a gain g₀ in the firstfiltering process F1 is fixed to “1.0” from a viewpoint of fixing thecenter of a distribution of frequency characteristics of a plurality oftypes of image filtering processes to stabilize the process.Accordingly, FIG. 25B shows gains g₀ to g₂ for realizing the respectivefrequency characteristics of the image filtering processes P1 to P4shown in FIG. 22 by a frequency characteristic of the entire systemshown in FIG. 25A with high accuracy in a case where a filter used ineach filter application unit 48 of the filtering process unit 41 havingthe circuit configuration shown in FIG. 11 is fixed and the gain g₀ inthe first filtering process F1 is fixed to “1.0”, for example.

As shown in FIG. 25B, according to the above-described example, it ispossible to compress data on the frequency characteristics of theplurality of types of image filtering processes P1 to P4 into frequencycharacteristics φ₀ to φ₂ (tap coefficients) of three filters and 12gains (8 gains in a case where g₀ is fixed to “1.0”).

In this way, the filtering process unit 41 in this example acquires thefilter application process data D4-1 using the frequency characteristicof the reference filter h_(b) determined according to the average of theplurality of types of frequency characteristics of the image filteringprocesses P1 to P4 as the “filter frequency characteristic φ₀” in thefirst filtering process F1, and acquires the filter application processdata D4-2 and D4-3 using the frequency characteristics of two variancefilters h_(v) determined according to the variance of the plurality oftypes of frequency characteristics of the image filtering processes P1to P4 as the “filter frequency characteristic φ₁” and the “filterfrequency characteristic φ₂” in the second filtering process F2 and thethird filtering process F3.

Particularly, it is preferable that the filtering process unit 41acquires the filter application process data D4-1 using “the referencefilter h_(b) determined according to the average of the frequencycharacteristics of the plurality of types of image filtering processesdetermined according to the optical characteristic of the optical systemaccording to the imaging condition when the original image data D1 isacquired” in the first filtering process F1, and that the filteringprocess unit 41 acquires the filter application process data D4-2 andD4-3 using “the variance filter determined according to the variance ofthe frequency characteristics of the plurality of types of imagefiltering processes determined according to the optical characteristicof the optical system according to the imaging condition when theoriginal image data D1 is acquired” in the second filtering process F2and the third filtering process F3. In this case, it is preferable thata gain applied to the filter application process data D4 in each of theplurality of times of filtering processes is determined based on theoptical characteristic of the optical system according to the imagingcondition when the original image data D1 is acquired.

FIG. 26 is a block diagram showing an example of a functionalconfiguration of the filter calculation unit 77 that calculates a filtercoefficient based on a unimodal distribution. In order to calculate afilter coefficient represented in an actual space from a filtercharacteristic represented in a frequency space, the filter calculationunit 77 in this example further includes a tap coefficient computationunit 81, in addition to the above-described average calculation unit 78,the variance calculation unit 79, and the filter characteristicacquisition unit 80 (see FIG. 19).

The tap coefficient computation unit 81 of the filter calculation unit77 acquires the reference filter h_(b) which has a plurality of taps, towhich a coefficient is allocated to each tap, and acquires at least onevariance filter which has a plurality of taps, to which a coefficient isallocated to each tap. A “method for allocating the coefficient to eachrespective tap” is not particularly limited, but the tap coefficientcomputation unit 81 (filter calculation unit 77) may calculate thecoefficient allocated to each tap of each of the reference filter h_(b)and the at least one variance filter h_(v), based on a SAGE algorithm oran OMP algorithm (which will be described later), for example.

<Multimodal Distribution>

Next, an example of a method for designing a filter and gain in a casewhere variation of frequency characteristics of a plurality of types ofimage filtering processes is considered as a multimodal distribution(see FIG. 21) will be described.

In the example shown in FIG. 21, in a case where the entire distributionis considered as one distribution (unimodal distribution), the variationdistribution of the frequency characteristics of the plurality of typesof image filtering processes can be represented by the “variationdistribution G” shown in FIG. 21, but approximation accuracy becomeslow. On the other hand, by separating the variation distribution of thefrequency characteristics of the plurality of types of image filteringprocesses into “the first variation distribution G1” and “the secondvariation distribution G2” in FIG. 21 and considering “the firstvariation distribution G1” and “the second variation distribution G2” asindividual multi-dimensional complex normal distributions (multimodaldistribution), it is possible to enhance the approximation accuracy. Asa parameter estimation algorithm of ML estimation in the multimodaldistribution, for example, estimation of a mixed normal distributionbased on an expectation maximization (EM) algorithm may be suitablyused, to thereby perform efficient parameter estimation.

The filter determination method based on the multimodal distribution maybe basically performed by the same method as the filter determinationmethod based on the above-described unimodal distribution. That is, itis possible to classify frequency characteristics of a plurality oftypes of image filtering processes into a plurality of distributionsthat forms the multimodal distribution (the first variation distributionG1 and the second variation distribution G2 in the example shown in FIG.21), to obtain a filter in the same procedure as that of “the variationdistribution G” of the above-described unimodal distribution withrespect to each distribution.

FIG. 27 is a diagram schematically showing an example of frequencycharacteristics of image filtering processes capable of being classifiedas a multimodal distribution, in which a lateral axis represents afrequency (ω) and a longitudinal axis represents a response.

FIG. 27 shows frequency characteristics of six types of image filteringprocesses (a first image filtering process P1, a second image filteringprocess P2, a third image filtering process P3, a fourth image filteringprocess P4, a fifth image filtering process P5, and a sixth imagefiltering process P6). The six types of image filtering processes P1 toP6 show different frequency characteristics, but in any case, a responsein a case where the frequency (ω) is “0” shows “1”.

In the example shown in FIG. 27, for example, it is possible to classify“frequency characteristic data of the first image filtering process P1,frequency characteristic data of the second image filtering process P2,and frequency characteristic data of the third image filtering processP3” into the first variation distribution G1, and to classify “frequencycharacteristic data of the fourth image filtering process P4, frequencycharacteristic data of the fifth image filtering process P5, andfrequency characteristic data of the sixth image filtering process P6”into the second variation distribution G2, for example.

FIGS. 28A and 28B are diagrams schematically showing an example of avariance of the frequency characteristics of the plurality of types ofimage filtering processes shown in FIG. 27, in which FIG. 28A shows avariance of the frequency characteristic data of the first imagefiltering process P1 to the third image filtering process P3 classifiedas the first variation distribution G1, and FIG. 28B shows a variance ofthe frequency characteristic data of the fourth image filtering processP4 to the sixth image filtering process P6 classified as the secondvariation distribution G2. A lateral axis in each of FIGS. 28A and 28Brepresents a frequency (ω), and a longitudinal axis represents avariance. FIGS. 28A and 28B do not strictly represent the variances ofthe frequency characteristics of the plurality of types of imagefiltering processes P1 to P6 shown in FIG. 27, for ease of description.

In any case of the frequency characteristics of the six image filteringprocesses P1 to P6 shown in FIG. 27, since the response in a case wherethe frequency (ω) is “0” is “1”, a variance in a case where thefrequency (ω) is “0” becomes “0”. As shown in FIGS. 28A and 28B, withrespect to “the variances of the frequency characteristics of the imagefiltering processes” of the first variation distribution G1 and thesecond variation distribution G2, respectively, “a component indicatingthe largest variation (see “H1” in FIGS. 28A and 28B)” and “a componentindicating the second largest variation (see “H2” in FIGS. 28A and 28B)”are shown.

FIGS. 29A to 29C are diagrams schematically showing examples of basefilters (characteristic vectors) acquired from an average and thevariance of the frequency characteristics of the first image filteringprocess P1 to the third image filtering process P3 classified as thefirst variation distribution G1, in which FIG. 29A shows “φ₀”, FIG. 29Bshows “φ₁”, and FIG. 29C shows φ₂″. FIGS. 30A to 30C are diagramsschematically showing examples of base filters (characteristic vectors)acquired from an average and the variance of the frequencycharacteristics of the fourth image filtering process P4 to the sixthimage filtering process P6 classified as the second variationdistribution G2, in which FIG. 30A shows “φ₀”, FIG. 30B shows φ₁″, andFIG. 30C shows φ₂″.

As described above, it is preferable that the frequency characteristicφ₀ of the filter used in the first filtering process F1 is determinedbased on the center (average) of a distribution of the frequencycharacteristics of the plurality of types of image filtering processes.

Further, it is preferable that the frequency characteristics of thefilters used in the other filtering processes are determined accordingto components of which the variations (variances) of the frequencycharacteristics of the plurality of types of image filtering processesshown in each of FIGS. 28A and 28B are large. Accordingly, it ispreferable that the frequency characteristic φ₁ of the filter used inthe second filtering process F2 is determined based on the componenthaving the largest variation of the frequency characteristics of theplurality of types of image filtering processes (see “H1” in FIGS. 28Aand 28B). Similarly, it is preferable that the frequency characteristicφ₂ of the filter used in the third filtering process F3 is determinedbased on the component having the second largest variation of thefrequency characteristics of the plurality of types of image filteringprocesses (see “H2” in FIGS. 28A and 28B).

FIGS. 31A and 31B show a frequency characteristic f(ω) (see FIG. 31A)and a gain example (see FIG. 31B) relating to the image filteringprocess P (first filtering process F1 to third filtering process F3)performed by the filtering process unit 41 having the circuitconfiguration shown in FIG. 11. In the frequency characteristic f(ω)relating to a multimodal distribution shown in FIGS. 31A and 31B, “φ₀^((i))(ω)” represents a frequency characteristic of a filter used in afirst filter application unit 48-1 of the first filtering process F1,and “g₀” represents a gain used in a first gain application unit 49-1 ofthe first filtering process F1. Further, “φ₁ ^((i))(ω)” represents afrequency characteristic of a filter used in a second filter applicationunit 48-2 of the second filtering process F2, and “g₁” represents a gainused in a second gain application unit 49-2 of the second filteringprocess F2. Furthermore, “φ₂ ^((i))(ω)” represents a frequencycharacteristic of a filter used in a third filter application unit 48-3of the third filtering process F3, and “g₂” represents a gain used in athird gain application unit 49-3 of the third filtering process F3.

The sign “i” in the frequency characteristics φ₁ ^((i))(ω) to φ₃^((i))(ω) of the respective filters represents a cluster indexindicating a group classified according to the multimodal distribution.For example, in the first image filtering process P1 to the third imagefiltering process P3 classified as the first variation distribution G1,“1” may be allocated as the cluster index i as shown in FIG. 31B. On theother hand, in the fourth image filtering process P4 to the sixth imagefiltering process P6 classified as the second variation distribution G2,“2” may be allocated as the cluster index i as shown in FIG. 31B.

FIG. 31B shows gains g₀ to g₂ for realizing respective frequencycharacteristics of the image filtering processes P1 to P6 shown in FIG.27 by a frequency characteristic of the entire system shown in FIG. 31Awith high accuracy in a case where a filter used in each filterapplication unit 48 of the filtering process unit 41 having the circuitconfiguration shown in FIG. 11 is fixed and the gain g₀ used in thefirst filtering process F1 is fixed to “1.0”, for example.

As shown in FIG. 31B, according to the above-described example, it ispossible to compress data on the frequency characteristics of theplurality of types of image filtering processes P1 to P6 into six filterfrequency characteristics φ₀ ⁽¹⁾ to φ₂ ⁽¹⁾, φ₀ ⁽²⁾ to φ₂ ⁽²⁾ (tapcoefficients) and 18 gains (12 gains in a case where g₀ is fixed to“1.0”).

In this way, according to the filter determination method based on themultimodal distribution, filter setting according to the number ofgroups that form the multimodal distribution (the first variationdistribution G1 and the second variation distribution G2 in the exampleshown in FIG. 21) is necessary. Accordingly, in a case where themultimodal distribution is used, compared with a case where a unimodaldistribution is used, the number of filters that needs to be stored in amemory in advance increases, and a necessary storage capacity alsoincreases. Further, in order to associate each filter with a group to beclassified, it is necessary to store association information such as theabove-described cluster index in a memory in advance.

FIG. 32 is a block diagram showing an example of a functionalconfiguration of the filter calculation unit 77 that calculates a filtercoefficient based on a multimodal distribution. The filter calculationunit 77 in this example further includes a filter group classificationunit 82, in addition to the average calculation unit 78, the variancecalculation unit 79, the filter characteristic acquisition unit 80 andthe tap coefficient computation unit 81 described above (see FIG. 26),in order to classify a plurality of types of frequency characteristicsof image filtering processes into a plurality of groups that form amultimodal distribution.

The filter group classification unit 82 in this example classifiesfrequency characteristics of a plurality of types of image filteringprocesses (in the example shown in FIG. 27, the first image filteringprocess P1 to the sixth image filtering process P6) into a plurality offilter groups (the first variation distribution G1 and the secondvariation distribution G2 in the example shown in FIG. 21) withreference to a mixed normal distribution.

FIG. 33 is a conceptual diagram of a plurality of types of imagefiltering processes (frequency characteristics) classified into aplurality of filter groups by the filter group classification unit 82.The filter group classification unit 82 analyzes similarity of frequencycharacteristic data of the plurality of types of image filteringprocesses, and classifies the frequency characteristics of therespective image filtering processes into corresponding filter groups.In the above-described example, the first image filtering process P1 tothe third image filtering process P3 are classified into a filter groupof the first variation distribution G1, and the fourth image filteringprocess P4 to the sixth image filtering process P6 are classified into afilter group of the second variation distribution G2, by the filtergroup classification unit 82.

The average calculation unit 78, the variance calculation unit 79, thefilter characteristic acquisition unit 80, and the tap coefficientcomputation unit 81 that form the filter calculation unit 77 acquire areference filter h_(b) and at least one variance filter h_(v) relatingto each of the plurality of filter groups, based on frequencycharacteristics of a plurality of types of image filtering processesincluded in each of the plurality of filter groups among the frequencycharacteristics of the plurality of types of image filtering processeswhich are classified into the plurality of filter groups.

According to the above-described example, in a case where a variationdistribution of frequency characteristics of a plurality of types ofimage filtering processes is already known, it is possible to design anoptimal filter for covering the variation with high accuracy.Particularly, by selecting an appropriate distribution type among aunimodal distribution and a multimodal distribution according to thevariation distribution of the frequency characteristics of the pluralityof types of image filtering processes, it is possible to acquire afilter (filter coefficient) and a gain capable of reproducing afrequency characteristic of a desired image filtering process with highaccuracy.

As described above, according to this embodiment, it is possible toreduce the number of filters (data amount), to prevent retention of FIRfilters (filter coefficients (tap coefficients)) relating to allconditions, and to perform an image filtering process using an averagelyexcellent filter.

<Design of FIR Filter>

Next, a method for designing an FIR filter for which there is a limit inthe number of taps will be described.

By flexibly adjusting a frequency characteristic of a filter by theabove-described method, it is possible to design an optimal filter foreffectively covering variation of frequency characteristics of aplurality of types of image filtering processes. However, in a casewhere a filter to be actually used is realized by an FIR filter forwhich there is a limit in the number of taps, it is difficult to ideallyobtain a desired frequency characteristic.

Generally, it is extremely difficult to realize a rapid frequencycharacteristic with a small number of taps. Particularly, acharacteristic vector corresponding to a small characteristic valuetends to have a rapid zigzag shape. Accordingly, even if an idealfrequency characteristic of a filter is acquired by the above-describedmethod, in a case where a sufficient number of taps of a filter to beactually used is not prepared, it is difficult to sufficiently covervariation of frequency characteristics of a plurality of types of imagefiltering processes.

This problem relates to a parametric function of which a base isnon-linear, and is similar to a problem that a base parameter isdetermined to approximate a certain vector by a linear sum of as a smallnumber of bases as possible. Various methods for solving such a problemhave been proposed. For example, a SAGE algorithm is an algorithm usedin incoming wave analysis in radio wave transmission, for example, andis a method for obtaining bases one by one to minimize a residual basedon a point of view of an EM algorithm. Further, an OMP algorithm is analgorithm used in a case where a sparse coefficient vector using anovercomplete dictionary is obtained in a compressed sensing filed. TheOMP method itself is not a method for obtaining a parametric base, butit is possible to apply the OMP method in design of an FIR filter bychanging a situation setting for selection from a dictionary in OMP to asituation setting for selection of a parametric base close to aparametric base function.

An example of a specific computation method of the SAGE algorithm andthe OMP algorithm will be described later, but the tap coefficientcomputation unit 81 (see FIGS. 26 and 32) of the filter calculation unit77 may calculate a coefficient allocated to each tap of at least onevariance filter h_(v) from a variation distribution function (see acovariance matrix “R₁” which will be described later) determined basedon a coefficient allocated to each tap of the reference filter h_(b),which is a variation distribution function indicating a variance offrequency characteristics of a plurality of types of image filteringprocesses, based on the algorithms.

According to the SAGE algorithm and the OMP algorithm (which will bedescribed later), the tap coefficient computation unit 81 of the filtercalculation unit 77 acquires, in a case where “I” is set to an integerwhich is equal to or greater than 2, a first to an I-th variance filtersh_(v), and calculates a coefficient allocated to each tap of the firstvariance filter h_(v) from a variation distribution function determinedbased on a coefficient allocated to each tap of the reference filterh_(b). Further, the tap coefficient computation unit 81 of the filtercalculation unit 77 calculates, in a case where “J” is set to an integerwhich is equal to or greater than 2 and is equal to or smaller than I, acoefficient allocated to each tap of a J-th variance filter h_(v), fromthe variation distribution function determined based on the coefficientallocated to each tap of the reference filter h_(b) and a coefficientallocated to each tap of the variance filter h_(v) which belongs to afirst to a (J−1)-th variance filters h_(v).

Further, in the SAGE algorithm, the tap coefficient computation unit 81of the filter calculation unit 77 updates a coefficient allocated toeach tap of at least one of the reference filter h_(b) and the at leastone variance filter h_(v), based on a variation distribution function(see a covariance matrix “Q_(j)” (which will be described later))indicating a residual component determined based on the coefficientallocated to each tap of the reference filter h_(b) and at least onevariance filter h_(v), which is a variation distribution functionindicating a variation of frequency characteristics of a plurality oftypes of image filtering processes.

The SAGE algorithm and the OMP algorithm are described in variousliteratures, and for example, may make reference to J. A. Fessler and A.O. Hero, “Space-Alternating Generalized Expectation-MaximizationAlgorithm,” IEEE Transactions on Signal Processing, Vol. 17, No. 10, pp.2664-2677, 1994; B. H. Fleury, M. Tschudin, R. Heddergott, D. Dahlhaus,and K. I. Pedersen, “Channel Parameter Estimation in Mobile RadioEnvironments Using the SAGE Algorithm,” IEEE J. Selected Areas inCommunications, Vol. 17, No. 3, pp. 434-449, Mar. 1999; and Y Pati, R.Rezaiifar, P. Krishnaprasad, “Orthogonal Matching Pursuit: recursivefunction approximation with application to wavelet decomposition”, inAsilomar Conf. on Signals, Systems and Comput., 1993. Here, since boththe SAGE algorithm and the OMP algorithm are provided for the purpose ofvector base decomposition and do not originally handle partial spaceapproximation in filter design, it is necessary to perform appropriatemodification with respect to a calculation expression.

<Calculation of Optimal Filter Group Based on Application of OMPAlgorithm>

Hereinafter, an example of filter design based on the OMP algorithm willbe described. Processes of respective steps described hereinbelow isbasically performed by the filter calculation unit 77 (tap coefficientcomputation unit 81), but a part or all of the processes of therespective steps may be performed by another unit.

Hereinafter, a case where a unimodal distribution is premised will bedescribed. With respect to a case where a multimodal distribution ispremised, as described above, a case where the unimodal distribution ispremised may be applied. That is, by handling each of a plurality offilter groups (in the example shown in FIG. 21, the first variationdistribution G1 and the second variation distribution G2) as thevariation distribution G of the unimodal distribution, it is possible tohandle a case where the multimodal distribution is premised.

Step 1

First, a tap coefficient of an FIR filter that most closely approximatesan average (a central value) of frequency characteristics of a pluralityof types of image filtering processes represented by the above-described“ψ₀” is acquired by an arbitrary method, and a frequency characteristicof a filter which is actually realized by the tap coefficient is set as“a frequency characteristic φ₀ of a filter used in a first filteringprocess F1.

Step 2

A covariance matrix R₀ in a case where “the frequency characteristic φ₀of the filter which is actually used in the first filtering process F1”is assumed as the average (central value) of the frequencycharacteristics of the plurality of types of image filtering processesis calculated based on the following expression.

$R_{0} = {\frac{1}{N_{p}}{\sum\limits_{i = 1}^{N_{p}}{\left( {\beta_{i} - \phi_{o}} \right)\left( {\beta_{i} - \phi_{o}} \right)^{H}}}}$

Step 3

The covariance matrix R₀ is set as a covariance matrix R₁, and theparameter “i” is set to “1” as shown in the following expression.R _(j) =R ₀i←1

Step 4

A characteristic vector ψ_(i) corresponding to a maximum characteristicvalue of the covariance matrix R₁ is obtained by the above-describedmethod.

Step 5

A tap coefficient of an FIR filter that most closely approximates afrequency characteristic of the characteristic vector ψ_(i) is acquiredby an arbitrary method, and a frequency characteristic of a filter whichis actually realized by the tap coefficient is set to “φ_(i)”. A filterhaving the frequency characteristic φ_(i) corresponds to a filter usedin an “i+1”-th filtering process, and for example, if “i=1”, a frequencycharacteristic φ₂ of a filter used in the second filtering process F2 isobtained. The tap coefficient of the filter relating to “φ_(i+1)” isstored in a memory (not shown).

Step 6

Using the already obtained “φ₁ to φ_(i)” as bases, a component capableof being represented by a partial space where the bases are spread isexcluded from the covariance matrix, and a residual is calculated. Forexample, if “φ_(i)=[φ₁ φ₂ . . . φ_(i)]” is defined by setting “i” to aninteger which is equal to or greater than 3, when a Moore-Penrosegeneralized inverse of φ_(i) is represented as “φ_(i) ⁺”, an orthogonalprojection matrix to a span {φ₁, φ₂, . . . φ_(i)} is represented as“φ_(i)φ_(i) ⁺”. If the orthogonal projection matrix φ_(i)φ_(i) ⁺ isused, the covariance matrix “R_(i+1)” is represented as the followingexpression.R _(i+1)=(I−Φ _(i)Φ_(i) ⁺)R ₀(I−Φ _(i)Φ_(i) ⁺)_(H)I: unit matrix

Step 7

As represented by the following expression, “i” is newly set to “i=i+1”,and then, “step 4” to “step 7” are repeated until “i=N−1” is satisfied.In this way, the filter frequency characteristics φ₁ to φ_(N-1) areobtained.i←i+1

<Calculation of Optimal Filter Group Based on Application of SAGEAlgorithm>

Next, an example of filter design based on the SAGE algorithm will bedescribed. Hereinafter, a case where a unimodal distribution is premisedwill be described, this example may be applied to a case where amultimodal distribution is premised, which is the same as theabove-described OMP algorithm.

The SAGE algorithm is different from the OMP algorithm in that updatingof the residual of the covariance matrix in the above-described step 6is simply based on subtraction of a selected base (FIR filter) and theupdating is individually continued with respect to each base even afterall bases are once obtained.

More specifically, step 1 to step 5 of the above-described OMP algorithmare similarly performed in the SAGE algorithm. Here, the covariancematrix “R_(i+1)” used in step 6 in the above-described OMP algorithm isobtained by the following expression in the SAGE algorithm.R _(i+1)=(1−φ_(i)φ_(i) ^(H))R _(i)(I−φ _(i)φ_(i) ^(H))^(H)

In step 7 of the above-described OMP algorithm is similarly performed inthe SAGE algorithm. Here, in the SAGE algorithm, step 8 to step 12 areperformed after step 7.

Step 8

As represented by the following expression, a parameter “j” is set to aninitial value “1”.j←1

Step 9

As represented by the following expression, a component capable of beingexpressed by a partial space where bases from which a j-th base isexcluded are spread is excluded from the covariance matrix, and aresidual is calculated.Ψ_(j)=[φ₁φ₂ . . . φ_(j−1)φ_(j+1) . . . φ_(N-1)]Q _(j)=(I−Ψ _(j)Ψ_(j) ⁺)R ₀(I−Ψ _(j)Ψ_(j) ⁺)^(H)

Step 10

A characteristic vector ψ_(j) corresponding to a maximum characteristicvalue of a covariance matrix Q indicating the residual componentrepresented by the above expression is obtained according to theabove-described method.

Step 11

A tap coefficient of an FIR filter that most closely approximates afrequency characteristic of the characteristic vector ψ_(j) is acquiredby an arbitrary method, and a frequency characteristic of a filter thatis actually realized by the tap coefficient is set to “φ_(j)”. Thus, thefrequency characteristic φ_(j)″ (tap coefficient) of the filter used ina “j+1”-th filtering process is updated, and the result is stored in amemory (not shown).

Step 12

Until an overall approximation error “J_(TOTAL)” defined as thefollowing expression is within a specific target range or the number ofloops (calculation time) reaches a specific upper limit, “step 9” to“step 12” are repeated while changing the parameter “j” as representedby the following expression.Φ_(ALL)=[φ₁φ₂ . . . φ_(N-1)]J _(TOTAL) =tr(I−Φ _(ALL)Φ_(ALL) ⁺)R ₀(I−Φ _(ALL)Φ_(ALL) ⁺)^(H)tr: trace (diagonal partial sum)

$\left. j\leftarrow\left\{ \begin{matrix}{j + 1} & {j < {N - 1}} \\1 & {otherwise}\end{matrix} \right. \right.$

By using the above-described SAGE algorithm or OMP algorithm, ifvariation (for example, average and variance) of frequencycharacteristics of a plurality of types of image filtering processes canbe calculated, it is possible to design a tap coefficient of an FIRfilter with high accuracy. Further, before calculation of a frequencycharacteristic of an individual specific image filtering process, it ispossible to determine a tap coefficient of a filter that can be actuallyused in an image filtering process (a plurality of filtering processes).

<Other Modification Examples>

The above-described respective functional configurations may be realizedby arbitrary hardware, software, or a combination of the hardware andsoftware. For example, the invention may be applied to a program thatcauses a computer to execute the image processing method (imageprocessing procedure and function), the gain acquisition method (gainacquisition processing procedure and function), and the filteracquisition method (filter acquisition processing procedure andfunction) in the above-described respective apparatuses and processingunits, a computer readable recording medium (non-transitory recordingmedium) in which the program is recorded, or a computer in which theprogram can be installed.

Further, the purpose of the image filtering process (the plurality oftimes of filtering processes) is not particularly limited, and in eachfiltering process (each filter application unit 48), various filterssuch as a restoration filter or a contour emphasis filter based on apoint spread function for the purpose of improving image quality, or anart filter or a diffusing filter for the purpose of providing a specialeffect, may be used.

<Application Example to EDoF System>

For example, the image filtering process according to the invention maybe applied to a restoration process with respect to image data (originalimage data D1) which is acquired through imaging by an optical system(an imaging lens or the like) having an extended depth of field (focus)(EDof). By performing the restoration process with respect to image dataof a blurred image which is acquired through imaging in a state wherethe EDof (focal depth) is extended by the EDof optical system, it ispossible to restore and generate image data with a high resolution in afocused state in a wide range. In this case, a restoration process usinga restoration filter which is based on an optical transfer function(PSF, OTF, MTF, PTF, or the like) of the EDoF optical system and has afilter coefficient set so that excellent image restoration can beperformed in the range of the extended depth of field (focal depth), maybe performed.

Hereinafter, an example of a system (EDoF system) relating torestoration of image data which is acquired through imaging using theEDoF optical system will be described. In the example shown hereinafter,an example in which a restoration process is performed with respect to abrightness signal (Y data) obtained from image data (RGB data) afterdemosaicing will be described, but a timing when the restoration processis performed is not particularly limited, and for example, therestoration process may be performed with respect to “image data (mosaicimage data) before the demosaicing” or “image data (demosaiced imagedata) after the demosaicing and before brightness signal conversion”.

FIG. 34 is a block diagram showing a form of an imaging module 101 thatincludes an EDoF optical system. The imaging module 101 (a digitalcamera or the like) in this example includes an EDoF optical system(lens unit) 110, an imaging element 112, an AD converter 114, and arestoration process block (image processing unit 35) 120.

FIG. 35 is a diagram showing an example of the EDoF optical system 110.The EDoF optical system 110 in this example includes fixed imaginglenses 110A having a single focus, and an optical filter 111 disposed ata pupil position. The optical filter 111 modulates a phase, and causesthe EDoF optical system 110 (imaging lenses 110A) to have EDoF so thatthe extended depth of field (focal depth) (EDoF) is obtained. In thisway, the imaging lenses 110A and the optical filter 111 form a lens unitthat modulates a phase to extend the depth of field.

The EDoF optical system 110 includes another component as necessary, andfor example, a diaphragm (not shown) is disposed in the vicinity of theoptical filter 111. Further, a single optical filter 111 may be used, ora combination of a plurality of optical filter 111 may be used.Furthermore, the optical filter 111 is only an example of optical phasemodulation means, and EDoF of the EDoF optical system 110 (imaging lens110A) may be realized by another means. For example, instead of theoptical filter 111, EDoF of the EDoF optical system 110 may be realizedby the imaging lens 110A lens-designed to have the same function as thatof the optical filter 111 in this example.

That is, EDoF of the EDoF optical system 110 may be realized by avariety of means for changing a wave front of image formation onto alight receiving surface of the imaging element 112. For example, “anoptical element of which a thickness is changed”, “an optical element ofwhich a refractive index is changed (a refractive index distributiontype wave front modulation lens, or the like)”, “an optical element ofwhich a thickness or a refractive index is changed due to coating on alens surface or the like (a wave front modulation hybrid lens, anoptical element formed on a lens surface as a phase surface, or thelike)”, or “a liquid crystal element capable of modulating a light phasedistribution (a liquid crystal space phase modulation element, or thelike)” may be employed as the EDoF means of the EDoF optical system 110.In this way, the invention is not only applied to a case where imageswhich are regularly dispersed by a light wave front modulation element(optical filter 111 (phase plate)) can be formed, but may also beapplied to a case where the same dispersed images as those formed usinga light wave front modulation element can be formed by the imaging lens110A itself without using the light wave front modulation element.

Since the EDoF optical system 110 shown in FIG. 35 may not be providedwith a focus adjustment mechanism that mechanically performs focusadjustment, it is possible to reduce its size, and thus, the EDoFoptical system 110 can be suitably mounted to a mobile phone or apersonal digital assistant with a camera.

An optical image after passing through the EDoF optical system 110having EDoF is formed on the imaging element 112 shown in FIG. 34, andis herein converted into an electric signal.

The imaging element 112 is configured by a plurality of pixels which arearranged in a matrix form using a predetermined pattern arrangement(Bayer arrangement, G-striped R/G full checker, X-Trans arrangement, orhoneycomb arrangement, or the like), and each pixel is configured toinclude a microlens, a color filter (in this example, an RGB colorfilter), and a photodiode. An optical image incident onto the lightreceiving surface of the imaging element 112 through the EDoF opticalsystem 110 is converted into an amount of signal electric chargescorresponding to the quantity depending on the intensity of incidentlight by each photodiode arranged on the light receiving surface.Further, R, G, and B signal electric charges accumulated in eachphotodiode are sequentially output as a voltage signal (image signal)for each pixel.

The AD converter 114 converts analog R, G, and B image signals outputfrom the imaging element 112 to each pixel into digital R, G, and Bimage signals. The digital image signals which are obtained byconversion into digital image signals in the AD converter 114 areapplied to the restoration process block 120.

The restoration process block 120 includes a black level adjustment unit122, a white balance gain unit 123, a gamma processing unit 124, ademosaicing unit 125, an RGB/YCrCb converter 126, and an Y signalrestoration processing unit 127.

The black level adjustment unit 122 performs black level adjustment withrespect to digital image signals output from the AD converter 114. Aknown method may be employed for the black level adjustment. Forexample, in a case where a certain effective photoelectric conversionelement is focused on, by calculating an average of dark amperageacquisition signals respectively corresponding to a plurality of OBphotoelectric conversion elements included in a photoelectric conversionelement line including the effective photoelectric conversion element,and by subtracting the average from the dark amperage acquisitionsignals corresponding to the effective photoelectric conversion element,it is possible to perform the black level adjustment.

The white balance gain unit 123 performs gain adjustment based on awhite balance gain of respective color signals of R, G, and B includedin the digital image signals for which the black level data is adjusted.

The gamma processing unit 124 performs gamma correction for performinggradation correction such as a half tone so that the R, G, and B imagesignals for which the white balance is adjusted have desired gammacharacteristics.

The demosaicing unit 125 performs demosaicing with respect to the R, G,and B image signals after gamma correction. Specifically, thedemosaicing unit 125 generates a set of image signals (R signal, Gsignal, and B signal) output from each light receiving pixel of theimaging element 112 by performing a color interpolation process withrespect to the R, G, and B image signals. That is, before demosaicing, apixel signal from each light receiving pixel is any one of the R, G, andB image signals, but after color demosaicing, a set of three pixelsignals of R, G, and B signals corresponding to each light receivingpixel is output.

The RGB/YCrCb converter 126 converts the R, G, and B signals for eachpixel, subjected to the demosaicing, into a brightness signal Y, and acolor difference signals Cr and Cb, and outputs the brightness signal Yand the color difference signals Cr and Cb for each pixel.

The Y signal restoration processing unit 127 performs a restorationprocess with respect to the brightness signal Y from the RGB/YCrCbconverter 126 based on a restoration filter which is stored in advance.The restoration filter is configured by a deconvolution kernel(corresponding to a tap number of M=7 and N=7) having a kernel size of7×7 and a computation coefficient (corresponding to restoration gaindata and a filter coefficient) corresponding to the deconvolutionkernel, and is used in a deconvolution process (deconvolutioncomputation process) corresponding to phase modulation of the opticalfilter 111. The restoration filter corresponds to the optical filter111, and is stored in a memory (not shown) (for example, a memory inwhich the Y signal restoration processing unit 127 is additionallyprovided). Further, the kernel size of the deconvolution kernel is notlimited to 7×7. The Y signal restoration processing unit 127 has afunction of a sharpening process in the above-described image processingunit 35.

Next, a restoration process in the restoration process block 120 will bedescribed. FIG. 36 is a diagram showing an example of a restorationprocess flow in the restoration process block 120 shown in FIG. 34.

Digital image signals from the AD converter 114 are applied to one inputend of the black level adjustment unit 122, and black level data isapplied to the other input end thereof. The black level adjustment unit122 subtracts the black level data from the digital image signals, andoutputs the digital image signals from which the black level data issubtracted to the white balance gain unit 123 (S41). Thus, a black levelcomponent is not included in the digital image signals, and thus, adigital image signal indicating the black level becomes 0.

The processes in the white balance gain unit 123 and the gammaprocessing unit 124 are sequentially performed with respect to imagedata after black level adjustment (S42 and S43).

The gamma-corrected R, G, and B signals are demosaiced in thedemosaicing unit 125, and then, are converted into the brightness signalY, and the color difference signals Cr and Cb in the RGB/YCrCb converter126 (S44).

The Y signal restoration processing unit 127 performs a restorationprocess for applying a deconvolution process corresponding to phasemodulation of the optical filter 111 of the EDoF optical system 110 tothe brightness signal Y (S45). That is, the Y signal restorationprocessing unit 127 performs a deconvolution process (deconvolutionoperating process) of brightness signals (herein, brightness signals of7×7 pixels) corresponding to a predetermined unit of pixel groupscentering around a pixel which is an arbitrary processing target andrestoration filters stored in advance in a memory or the like (7×7deconvolution kernels and their computation coefficients). The Y signalrestoration processing unit 127 performs a restoration process forremoving image blurring of an entire image by repeating the convolutionprocess for each of the predetermined unit of pixel groups to cover anentire area of an imaging surface. The restoration filter is determinedaccording to the position of the center of the pixel groups for whichthe deconvolution process is performed. That is, a common restorationfilter is applied to contiguous pixel groups. Further, in order tosimplify the restoration process, it is preferable to apply a commonrestoration filter to all the pixel groups.

As shown in (a) of FIG. 37, a point image (optical image) of brightnesssignals after passing through the EDoF optical system 110 are formed onthe imaging element 112 as a large point image (blurred image), but isrestored as a small point image (high-resolution image) by thedeconvolution process in the Y signal restoration processing unit 127,as shown in (b) of FIG. 37.

As described above, by applying a restoration process to brightnesssignals after demosaicing, it is not necessary to separately provideparameters of the restoration process for each of R, G, and B, and thus,it is possible to perform the restoration process at high speed.Further, since respective image signals of R, G, and B corresponding topixels of R, G, and B disposed at scattered positions are not collectedin individual units to perform the deconvolution process, but instead,brightness signals of contiguous pixels are collected in a predeterminedunit and a common restoration filter is applied to the unit to performthe deconvolution process, the accuracy of the restoration process isenhanced. With respect to the color difference signals Cr and Cb, eventhough the resolution is not increased by the restoration process interms of visual characteristics of human's eyes, an image may be allowedin terms of image quality. Further, in a case where an image is recordedin a compression format such as JPEG, since color difference signals arecompressed at a high compression rate compared with that of a brightnesssignal, it is not necessary to increase the resolution by therestoration process. Thus, it is possible to enhance the accuracy ofrestoration, and simultaneously, to achieve simplification and speedincrease of the process.

With respect to the restoration process of the above-described EDoFsystem, similarly, it is possible to apply the image filtering processaccording to the above-described embodiment.

Further, a form to which the invention is applicable is not limited to adigital camera and a computer (server). For example, the invention maybe applied to various types of cameras that have an imaging function asa main function, and also, may be applied to various types of mobiledevices that have other functions (call function, communicationfunction, and other computer functions) in addition to an imagingfunction. As other forms to which the invention is applicable, forexample, a mobile phone, a smartphone, a personal digital assistant(PDA) and a mobile game machine having a camera function may be used.Hereinafter, an example of a smartphone to which the invention isapplicable will be described.

<Application Example to Smartphone>

FIG. 38 is a diagram showing an appearance of a smartphone 201. Thesmartphone 201 shown in FIG. 38 includes a flat housing 202, and adisplay input unit 220 that is disposed on one surface of the housing202 and includes a display panel 221 which is a display unit and anoperation panel 222 which is an input unit, in which the display panel221 and the operation panel 222 are integrally formed. Further, thehousing 202 includes a speaker 231, a microphone 232, an operation unit240, and camera unit 241. The configuration of the housing 202 is notlimited to thereto, and for example, a configuration in which a displayunit and an input unit are provided in an independent manner may beemployed, or a configuration in which a folding structure or a slidemechanism is provided may also be employed.

FIG. 39 is a block diagram showing a configuration of the smartphone 201shown in FIG. 38. As shown in FIG. 39, as main components of thesmartphone, a wireless communication unit 210, the display input unit220, a call unit 230, the operation unit 240, the camera unit 241, astorage unit 250, an external input/output unit 260, a globalpositioning system (GPS) receiver unit 270, a motion sensor unit 280, apower source unit 290, and a main controller 200 (including theabove-described main body controller 25) are provided. Further, as mainfunctions of the smartphone 201, a wireless communication function forperforming mobile wireless communication with a base station device BSthrough a mobile communication network NW.

The wireless communication unit 210 performs wireless communication withthe base station device BS connected to the mobile communication networkNW according to an instruction of the main controller 200. Using thewireless communication, the wireless communication unit 210 performstransmission and reception of a variety of file data such as sound dataor image data, e-mail data, or the like, and performs reception of Webdata, streaming data, or the like.

The display input unit 220 is a so-called touch panel that includes adisplay panel 221 and an operation panel 222, displays an image (a stillimage and a moving image), character information, or the like under thecontrol of the main controller 200 to visually transmit information to auser, and detects a user operation with respect to the displayedinformation.

The display panel 221 uses a liquid crystal display (LCD), an organicelectro-luminescence display (OELD), or the like as a display device.The operation panel 222 is a device that is provided so that an imagedisplayed on a display surface of the display panel 221 can be visuallyrecognized and detects one or a plurality of sets of coordinatesoperated by a user's finger or a stylus. If the device is operated by auser's finger or a stylus, the operation panel 222 outputs a detectionsignal generated due to the operation to the main controller 200. Then,the main controller 200 detects an operation position (coordinates) onthe display panel 221 based on the received detection signal.

As shown in FIG. 38, as an embodiment of the imaging device of theinvention, a configuration in which the display panel 221 and theoperation panel 222 of the smartphone 201 shown as an example in FIG. 38are integrated to form the display input unit 220 and the operationpanel 222 is disposed to completely cover the display panel 221 may beused. In a case where such a configuration is employed, the operationpanel 222 may have a function of detecting a user operation in a regionout of the display panel 221. In other words, the operation panel 222may include a detection region with respect to a portion that overlapsthe display panel 221 (hereinafter, referred to as a “display region”),and a detection region with respect to an outer edge portion that doesnot overlap the display panel 221 (hereinafter, referred to as a“non-display region”).

The size of the display region and the size of the display panel 221 maybe completely the same, but it is not essential that both of the sizesare the same. Further, the operation panel 222 may include two sensitiveregions of an outer edge portion and an inner portion other than theouter edge portion. Further, the width of the outer edge portion isappropriately designed according to the size of the housing 202, or thelike. Furthermore, as a position detection method employed in theoperation panel 222, any one of a matrix switch type, a resistive filmtype, a surface acoustic wave type, an infrared type, an electromagneticinduction type, an electrostatic capacitance type, and the like may beemployed.

The call unit 230 includes the speaker 231 and the microphone 232. Thecall unit 230 converts user's voice input through the microphone 232into voice data capable of being processed by the main controller 200and outputs the result to the main controller 200, or decodes voice datareceived by the wireless communication unit 210 or the externalinput/output unit 260 and outputs the result through the speaker 231.Further, as shown in FIG. 38, for example, the speaker 231 may bemounted on the same surface as the surface where the display input unit220 is provided, and the microphone 232 may be mounted on a side surfaceof the housing 202.

The operation unit 240 is a hardware key using a key switch or the like,and receives an instruction from a user. For example, as shown in FIG.38, the operation unit 240 is a push button switch that is mounted on aside surface of the housing 202 of the smartphone 201, is turned on whenbeing pressed by a finger or the like, and is turned off by a restoringforce of a spring or the like when the finger is separated.

The storage unit 250 stores a control program or control data of themain controller 200, application software, address data in which a name,a telephone number, and the like of a communication partner areassociated with each other, data on transmitted or received e-mails, Webdata downloaded by a Web browser, or data on downloaded content, andtemporarily stores streaming data or the like. Further, the storage unit250 includes an internal storage unit 251 provided in the smartphone,and an external storage unit 252 provided with a slot for a detachableexternal memory. Each of the internal storage unit 251 and the externalstorage unit 252 that form the storage unit 250 is realized using astorage medium such as a flash memory, a hard disk, a multimedia cardmicro type memory, a card type memory (for example, MicroSD (registeredtrademark) memory or the like), a random access memory (RAM), a readonly memory (ROM), or the like.

The external input/output unit 260 serves as an interface with respectto all types of external devices to be connected to the smartphone 201,and is directly or indirectly connected to other external devicesthrough communication or the like (for example, Universal Serial Bus(USB), IEEE1394, or the like) or a network (for example, Internet,wireless LAN, Bluetooth (registered trademark), Radio FrequencyIdentification (RFID), Infrared Data Association (IrDA, registeredtrademark), Ultra Wideband (UWB, registered trademark), ZigBee(registered trademark), or the like).

As an external device to be connected to the smartphone 201, forexample, a wired or wireless headset, a wired or wireless externalcharger, a wired or wireless data port, a memory card, a SubscriberIdentity Module (SIM) card or a User Identity Module (UIM) cardconnected through a card socket, an external audio/video deviceconnected through an audio/video input/output (I/O) terminal, anexternal audio/video device connected in a wireless manner, a smartphoneconnected in a wired or wireless manner, a personal computer connectedin a wired or wireless manner, a PDA connected in a wired or wirelessmanner, an earphone or the like connected in a wired or wireless manner,may be used. The external input/output unit may be configured totransfer received data transmitted from the external device torespective components in the smartphone 201, or to transmit data in thesmartphone 201 to the external device.

The GPS receiver unit 270 receives GPS signals transmitted from GPSsatellites ST1 to STn according to an instruction of the main controller200, executes a positioning computation process based on the pluralityof received GPS signals, and detects a position specified by thelatitude, longitude and altitude of the smartphone 201. In a case whereposition information can be acquired from the wireless communicationunit 210 and/or the external input/output unit 260 (for example,wireless LAN), the GPS receiver unit 270 can also detect the positionusing the position information.

The motion sensor unit 280 includes a triaxial acceleration sensor orthe like, for example, and detects a physical movement of the smartphone201 according to an instruction of the main controller 200. By detectingthe physical movement of the smartphone 201, a direction and anacceleration where the smartphone 201 moves are detected. The detectionresult is output to the main controller 200.

The power source unit 290 supplies power accumulated in a battery (notshown) to respective units of the smartphone 201 according to aninstruction of the main controller 200.

The main controller 200 includes a microprocessor, and is operatedaccording to a control program or control data stored in the storageunit 250 to generally control the respective units of the smartphone201. Further, the main controller 200 has a mobile communication controlfunction for controlling respective units of a communication system andan application processing function in order to perform voicecommunication or data communication through the wireless communicationunit 210.

The application processing function is realized as the main controller200 is operated according to application software stored in the storageunit 250. As the application processing function, for example, aninfrared communication function for controlling the externalinput/output unit 260 to perform data communication with an opposingdevice, an e-mail function for performing transmission and reception ofe-mails, a Web browsing function for browsing Web pages, or the like isused.

Further, the main controller 200 has an image processing function, forexample, for displaying video on the display input unit 220 based onimage data (data on a still image or a moving image) such as receiveddata or downloaded streaming data. The image processing function refersto a function for decoding the image data, performing image processingwith respect to the decoded image data, and displaying an image obtainedthrough the image processing on the display input unit 220, by the maincontroller 200.

In addition, the main controller 200 executes a display control withrespect to the display panel 221, and an operation detection control fordetecting a user operation through the operation unit 240 or theoperation panel 222.

By executing the display control, the main controller 200 displays anicon for starting up application software or a software key such as ascroll bar, or displays a window for creating an e-mail. The scroll barrefers to a software key for receiving, with respect to a large imagewhich cannot be accommodated in a display region of the display panel221, an instruction for movement of a display portion of the image.

Further, by execution of the operation detection control, the maincontroller 200 detects a user operation through the operation unit 240,receives an operation with respect to the icon or an input of acharacter string with respect to an input section of the window, throughthe operation panel 222, or receives a scroll request of a display imagethrough the scroll bar.

Furthermore, by execution of the operation detection control, the maincontroller 200 includes a touch panel control function for determiningwhether an operation position with respect to the operation panel 222corresponds to a portion (display region) that overlaps the displaypanel 221 or an outer edge portion (non-display region) that does notoverlap the display panel 221, and controlling a sensitive region of theoperation panel 222 and a display position of a software key.

The main controller 200 may detect a gesture operation with respect tothe operation panel 222, and may execute a predetermined functionaccording to the detected gesture operation. The gesture operation doesnot refer to a typical simple operation, but refers to an operation ofdrawing a locus using a finger or the like, an operation ofsimultaneously designating a plurality of positions, or an operation ofdrawing a locus with respect to at least one of a plurality of positionsby combination of the above operations.

The camera unit 241 is a digital camera that performs electronic imagingusing an imaging element such as a CMOS or a CCD. The camera unit 241may convert image data obtained through imaging into compressed imagedata such as JPEG, for example, may record the image data in the storageunit 250, or may output the image data through the external input/outputunit 260 or the wireless communication unit 210, under the control ofthe main controller 200. In the smartphone 201 shown in FIG. 38, thecamera unit 241 is mounted on the same surface as that of the displayinput unit 220, but the mounting position of the camera unit 241 is notlimited thereto, and the camera unit 241 may be mounted on a rearsurface of the display input unit 220, or a plurality of camera units241 may be mounted on the display input unit 220. In a case where theplurality of camera units 241 are mounted, imaging may be performedusing a single camera unit 241 while switching the camera units 241 tobe provided for imaging, or may be performed using the plurality ofcamera units 241 at the same time.

Further, the camera unit 241 may be used for various functions of thesmartphone 201. For example, an image acquired by the camera unit 241may be displayed on the display panel 221, or an image in the cameraunit 241 may be used as one of operation inputs through the operationpanel 222. Further, when detecting a position using the GPS receiverunit 270, the position may be detected with reference to an image fromthe camera unit 241. In addition, it is possible to determine an opticalaxis direction or a current usage environment of the camera unit 241 ofthe smartphone 201 without using a triaxial acceleration sensor or byusing the triaxial acceleration sensor together, with reference to theimage from the camera unit 241. Furthermore, the image from the cameraunit 241 may be used in the application software.

Furthermore, position information acquired by the GPS receiver unit 270,voice information (which may be text information obtained by performingvoice text conversion by the main controller or the like) acquired bythe microphone 232, posture information acquired by the motion sensorunit 280, or the like may be added to image data on a still image or amoving image, and the result may be recorded in the storage unit 250, ormay be output through the external input/output unit 260 or the wirelesscommunication unit 210.

The above-described image processing unit 35 (filtering process unit 41)may be realized by the main controller 200, for example.

The invention is not limited to the above-described embodiments, andvarious modifications may be made in a range without departing from theconcept of the invention.

EXPLANATION OF REFERENCES

-   -   10: digital camera    -   11: lens unit    -   12: camera main body    -   14: lens    -   15: diaphragm    -   17: optical system operation unit    -   18: lens unit controller    -   19: lens unit storage unit    -   20: lens unit input/output unit    -   24: imaging element    -   25: main body controller    -   26: user interface    -   27: display unit    -   28: camera main body input/output unit    -   29: main body storage unit    -   30: input/output interface    -   34: device controller    -   35: image processing unit    -   36: display controller    -   40: pre-processing unit    -   41: filtering process unit    -   42: post-processing unit    -   43: gain specifying unit    -   44: gain candidate data storage unit    -   45: gain acquisition unit    -   48: filter application unit    -   49: gain application unit    -   50: process data calculation unit    -   51: processed image data calculation unit    -   52: adder unit    -   54: repetitive computation determination unit    -   55: gain supply unit    -   56: filter supply unit    -   60: reference image acquisition unit    -   61: reference image analysis unit    -   62: target frequency characteristic acquisition unit    -   63: application gain calculation unit    -   66: frequency analysis unit    -   67: target frequency characteristic specifying unit    -   68: frequency characteristic storage unit    -   76: filter acquisition apparatus    -   77: filter calculation unit    -   78: average calculation unit    -   79: variance calculation unit    -   80: filter characteristic acquisition unit    -   81: tap coefficient computation unit    -   82: filter group classification unit    -   92: computer    -   93: computer input/output unit    -   94: computer controller    -   95: display    -   96: network    -   97: server    -   98: server input/output unit    -   99: server controller    -   101: imaging module    -   110: EDoF optical system    -   110A: imaging lens    -   111: optical filter    -   112: imaging element    -   114: AD converter    -   120: restoration process block    -   122: black level adjustment unit    -   123: white balance gain unit    -   124: gamma processing unit    -   125: demosaicing unit    -   126: YCrCb converter    -   127: Y signal restoration processing unit    -   200: main controller    -   201: smartphone    -   202: housing    -   210: wireless communication unit    -   220: display input unit    -   221: display panel    -   222: operation panel    -   230: call unit    -   231: speaker    -   232: microphone    -   240: operation unit    -   241: camera unit    -   250: storage unit    -   251: internal storage unit    -   252: external storage unit    -   260: external input/output unit    -   270: GPS receiver unit    -   280: motion sensor unit    -   290: power source unit

What is claimed is:
 1. An image processing apparatus comprising: afiltering process circuit that performs an image filtering process thatincludes a plurality of times of filtering processes with respect tooriginal image data to acquire processed image data, wherein in each ofthe plurality of times of filtering processes, the filtering processcircuit applies a filter to processing target data to acquire filterapplication process data, applies a gain to the filter applicationprocess data to acquire gain application process data, and acquiresfiltering process data from the gain application process data, andwherein in each of the plurality of times of filtering processes, thegain applied to the filter application process data is acquired based ona target frequency characteristic of the image filtering processspecified based on an optical characteristic of an optical system usedwhen the original image data is acquired, that is, an opticalcharacteristic according to an imaging condition when the original imagedata is acquired.
 2. The image processing apparatus according to claim1, further comprising: a gain candidate data storage that stores gaintable information obtained by associating candidate data of the gainapplied to the filter application process data with the imagingcondition, in each of the plurality of times of filtering processes; anda gain specifying circuit that specifies, with reference to the gaintable information, the candidate data associated with the imagingcondition when the original image data is acquired as the gain appliedto the filter application process data in each of the plurality of timesof filtering processes, wherein the filtering process circuit appliesthe gain specified by the gain specifying circuit to the filterapplication process data to acquire the gain application process data ineach of the plurality of times of filtering processes.
 3. The imageprocessing apparatus according to claim 1, wherein the filter isdetermined according to the optical characteristic of the opticalsystem.
 4. The image processing apparatus according to claim 3, whereinthe filter is determined according to the optical characteristic of theoptical system under a representative imaging condition.
 5. The imageprocessing apparatus according to claim 4, wherein the representativeimaging condition includes a plurality of imaging conditions in whichfirst index conditions are different, that is, a plurality of imagingconditions based on the discontinuous first index conditions among theplurality of first index conditions capable of being employed in imagingof the original image data.
 6. The image processing apparatus accordingto claim 1, wherein the optical characteristic is based on the pointspread function.
 7. The image processing apparatus according to claim 1,wherein the gain is acquired by fitting a frequency characteristic ofthe image filtering process to the target frequency characteristic ofthe image filtering process using a least squares method based on eachfrequency characteristic of the plurality of times of filteringprocesses.
 8. The image processing apparatus according to claim 7,wherein weighting is performed based on a frequency in the least squaresmethod.
 9. The image processing apparatus according to claim 8, whereina weight in a low-frequency band is set to be larger than a weight in ahigh-frequency band in the least squares method.
 10. The imageprocessing apparatus according to claim 8, wherein a weight in ahigh-frequency band is set to be larger than a weight in a low-frequencyband according to an imaging condition when the original image data isacquired, in the least squares method.
 11. The image processingapparatus according to claim 8, wherein the weight in the least squaresmethod is determined according to the pixel position in the originalimage data.
 12. The image processing apparatus according to claim 11,wherein the weight in the high-frequency band is large at a pixelposition which is equal to or shorter than a first distance from thecenter of an image of the original image data, compared with a pixelposition which is more distant than the first distance from the centerof the image of the original image data, in the least squares method.13. The image processing apparatus according to claim 11, wherein theweight in the low-frequency band is large at a pixel position which ismore distant than a second distance from the center of an image of theoriginal image data, compared with a pixel position which is equal to orshorter than the second distance from the center of the image of theoriginal image data, in the least squares method.
 14. The imageprocessing apparatus according to claim 8, wherein the filtering processcircuit uses a filter that makes the filtering process data equal to theprocessing target data in each of the plurality of times of filteringprocesses at a frequency where a ratio of the processed image data tothe original image data is smaller than 1 in the target frequencycharacteristic of the image filtering process.
 15. The image processingapparatus according to claim 1, wherein the filtering process circuitacquires the filter application process data using a filter having afrequency characteristic according to a pixel position in the processingtarget data, in at least any one filtering process among the pluralityof times of filtering processes.
 16. The image processing apparatusaccording to claim 1, wherein the plurality of times of filteringprocesses include at least a first filtering process and a secondfiltering process, and wherein the filtering process circuit uses thefiltering process data acquired in the first filtering process as theprocessing target data in the second filtering process.
 17. The imageprocessing apparatus according to claim 1, wherein the plurality oftimes of filtering processes include at least a first filtering processand a second filtering process, and wherein the filtering processcircuit uses the same data in the first filtering process and the secondfiltering process as the processing target data, and acquires theprocessed image data based on the filtering process data acquired by thefirst filtering process and the filtering process data acquired by thesecond filtering process.
 18. The image processing apparatus accordingto claim 1, wherein the plurality of times of filtering processesinclude at least a first filtering process and a second filteringprocess, and wherein the filtering process circuit includes a firstfilter application circuit that applies a filter for the first filteringprocess to the processing target data of the first filtering process toacquire the filter application process data, a first gain applicationcircuit that applies a gain for the first filtering process to thefilter application process data acquired by the first filter applicationcircuit to acquire the gain application process data, a second filterapplication circuit that applies a filter for the second filteringprocess to the processing target data of the second filtering process toacquire the filter application process data, and a second gainapplication circuit that applies a gain for the second filtering processto the filter application process data acquired by the second filterapplication circuit to acquire the gain application process data. 19.The image processing apparatus according to claim 1, wherein theplurality of times of filtering processes include at least a firstfiltering process and a second filtering process, wherein the filteringprocess circuit includes a filter application circuit that applies thefilter to the processing target data to acquire the filter applicationprocess data, and a gain application circuit that applies the gain tothe filter application process data to acquire the gain applicationprocess data, wherein the filter application circuit acquires the filterapplication process data using a filter for the first filtering processin the first filtering process, and acquires the filter applicationprocess data using a filter for the second filtering process in thesecond filtering process, and wherein the gain application circuitacquires the gain application process data using a gain for the firstfiltering process in the first filtering process, and acquires the gainapplication process data using a gain for the second filtering processin the second filtering process.
 20. The image processing apparatusaccording to claim 1, wherein the plurality of times of filteringprocesses include at least a first filtering process and a secondfiltering process, and wherein the filtering process circuit acquiresthe filter application process data using a reference filter determinedaccording to an average of a plurality of types of frequencycharacteristics of the image filtering processes in the first filteringprocess, and acquires the filter application process data using avariance filter determined according to a variance of the plurality oftypes of frequency characteristics of the image filtering processes inthe second filtering process.
 21. An image processing apparatuscomprising: a filtering process circuit that performs an image filteringprocess including a plurality of times of filtering processes withrespect to original image data to acquire processed image data, whereinthe plurality of times of filtering processes include at least a firstfiltering process and a second filtering process, wherein the filteringprocess circuit applies a filter to a processing target data to acquirefilter application process data, applies a gain to the filterapplication process data to acquire gain application process data, andacquires filtering process data from the gain application process data,in each of the plurality of times of filtering processes, acquires, inthe first filtering process, the filter application process data using areference filter determined according to an average of frequencycharacteristics of the plurality of types of image filtering processes,which are frequency characteristics of the plurality of types of imagefiltering processes determined according to the optical characteristicof the optical system according to an imaging condition when theoriginal image data is acquired, acquires, in the second filteringprocess, the filter application process data using a variance filterdetermined according to a variance of the frequency characteristics ofthe plurality of types of image filtering processes, which are thefrequency characteristics of the plurality of types of image filteringprocesses determined according to the optical characteristic of theoptical system according to an imaging condition when the original imagedata is acquired, and wherein the gain applied to the filter applicationprocess data is determined based on the optical characteristic of theoptical system according to an imaging condition when the original imagedata is acquired, in each of the plurality of times of filteringprocesses.
 22. An image processing method for performing an imagefiltering process including a plurality of times of filtering processeswith respect to original image data to acquire processed image data,comprising: applying a filter to processing target data to acquirefilter application process data, applying a gain to the filterapplication process data to acquire gain application process data, andacquiring filtering process data from the gain application process data,in each of the plurality of times of filtering processes, and acquiringthe gain applied to the filter application process data based on atarget frequency characteristic of the image filtering process specifiedbased on an optical characteristic of an optical system used when theoriginal image data is acquired, that is, the optical characteristicaccording to an imaging condition when the original image data isacquired, in each of the plurality of times of filtering processes. 23.An image processing method for performing an image filtering processincluding a plurality of times of filtering processes with respect tooriginal image data to acquire processed image data, wherein theplurality of times of filtering processes include at least a firstfiltering process and a second filtering process, wherein the methodcomprises: applying a filter to processing target data to acquire filterapplication process data, applying a gain to the filter applicationprocess data to acquire gain application process data, and acquiringfiltering process data from the gain application process data, in eachof the plurality of times of filtering processes; acquiring, in thefirst filtering process, the filter application process data using areference filter determined according to an average of frequencycharacteristics of a plurality of types of image filtering processes,which are frequency characteristics of the plurality of types of imagefiltering processes determined according to the optical characteristicof the optical system according to an imaging condition when theoriginal image data is acquired; and acquiring, in the second filteringprocess, the filter application process data using a variance filterdetermined according to a variance of the frequency characteristics ofthe plurality of types of image filtering processes, which are thefrequency characteristics of the plurality of types of image filteringprocesses determined according to the optical characteristic of theoptical system according to an imaging condition when the original imagedata is acquired, and wherein the gain applied to the filter applicationprocess data is determined based on the optical characteristic of theoptical system according to an imaging condition when the original imagedata is acquired, in each of the plurality of times of filteringprocesses.
 24. A non-transitory computer-readable recording medium thatstores a program that causes a computer to realize a function forperforming an image filtering process including a plurality of times offiltering processes with respect to original image data to acquireprocessed image data, the non-transitory computer-readable recordingmedium that stores the program causing the computer to execute: aprocess of applying a filter to processing target data to acquire filterapplication process data, a process of applying a gain to the filterapplication process data to acquire gain application process data, and aprocess of acquiring filtering process data from the gain applicationprocess data, in each of the plurality of times of filtering processes,wherein the gain applied to the filter application process data isacquired based on a target frequency characteristic of the imagefiltering process specified based on an optical characteristic of anoptical system used when the original image data is acquired, that is,the optical characteristic according to an imaging condition when theoriginal image data is acquired, in each of the plurality of times offiltering processes.
 25. A non-transitory computer-readable recordingmedium that stores a program that causes a computer to realize afunction for performing an image filtering process including a pluralityof times of filtering processes with respect to original image data toacquire processed image data, wherein the plurality of times offiltering processes include at least a first filtering process and asecond filtering process, wherein the non-transitory computer-readablerecording medium that stores the program causes the computer to execute:a process of applying a filter to processing target data to acquirefilter application process data, a process of applying a gain to thefilter application process data to acquire gain application processdata, and a process of acquiring filtering process data from the gainapplication process data, in each of the plurality of times of filteringprocesses; a process of acquiring, in the first filtering process, thefilter application process data using a reference filter determinedaccording to an average of frequency characteristics of the plurality oftypes of image filtering processes, which are the frequencycharacteristics of the plurality of types of image filtering processesdetermined according to the optical characteristic of the optical systemaccording to an imaging condition when the original image data isacquired; and a process of acquiring, in the second filtering process,the filter application process data using a variance filter determinedaccording to a variance of the frequency characteristics of theplurality of types of image filtering processes, which are the frequencycharacteristics of the plurality of types of image filtering processesdetermined according to the optical characteristic of the optical systemaccording to an imaging condition when the original image data isacquired, and wherein the gain applied to the filter application processdata is determined based on the optical characteristic of the opticalsystem according to an imaging condition when the original image data isacquired, in each of the plurality of times of filtering processes.